Varnish

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents.

Setup

The varnish plugin is included with the AppOptics host agent by default, please follow the directions below to enable it for a host agent instance.

Prerequisites

The Varnish daemon must be started beforehand and configured to point to your HTTP service. By default, the Varnish service will run on port 6081, and the configuration file at /etc/varnish/default.vcl will need to be edited to point to your HTTP service.

Once the configuration file has been updated, reload the service:

$ service varnish reload

If you updated the service file to change the port Varnish runs on, you will have to reload the service file:

$ systemctl daemon-reload

Please visit the Varnish website for more information.

Configuration

The host agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent. To enable the plugin:

  1. Make a copy of the Varnish example configuration file /opt/appoptics/etc/plugins.d/varnish.yaml.example, renaming it to /opt/appoptics/etc/plugins.d/varnish.yaml:
$ sudo cp /opt/appoptics/etc/plugins.d/varnish.yaml.example /opt/appoptics/etc/plugins.d/varnish.yaml
  1. Update the /opt/appoptics/etc/plugins.d/varnish.yaml configuration file to set which metrics to report and to ensure varnishstat is executable if needed, for example:
collector:
  varnish:
    all:
      ## If running as a restricted user you can prepend sudo for additional access:
      # use_sudo: false
      ## The default location of the varnishstat binary can be overridden with:
      # binary: "/usr/bin/varnishstat"
      ## By default, telegraf gather stats for 3 metric points.
      ## Setting stats will override the defaults shown below.
      ## Glob matching can be used, ie, stats: "MAIN.*"
      ## stats may also be set to "*", which will collect all stats
      stats: "MAIN.backend_*,MAIN.cache_*,MAIN.client_req,MAIN.fetch_*,MAIN.losthdr,MAIN.n_expired,
              MAIN.n_lru_*,MAIN.s_re*,MAIN.s_sess,MAIN.sess_*,MAIN.shm_*,MAIN.thread*,MAIN.uptime"

load:
  plugin: snap-plugin-collector-bridge-varnish
  task: task-bridge-varnish.yaml
  1. Restart the host agent:
$ sudo service appoptics-snapteld restart
  1. Enable the Varnish plugin in AppOptics

    On the Integrations Page you will see the Varnish plugin available if the previous steps were successful. If you do not see the plugin, see Troubleshooting.

    Select the Varnish plugin to open the configuration menu in the UI, and enable the plugin.

Permission issues with running varnishstat

If you’re seeing errors related to running varnishstat, you may need to set additional permissions for the appoptics user.

  1. First, try adding the appoptics user to the varnish group:

    $ usermod -a -G varnish appoptics
    

    Confirm that the user has been updated:

    $ groups appoptics
    appoptics : appoptics varnish
    

    Then restart the service:

    $ sudo service appoptics-snapteld restart
    
  2. If that doesn’t work, try giving the appoptics user sudo privileges:

    Set use_sudo to true in the /opt/appoptics/etc/plugins.d/varnish.yaml file.

    ## If running as a restricted user you can prepend sudo for additional access:
    use_sudo: true
    

    Then restart the service:

    $ sudo service appoptics-snapteld restart
    

Default Metrics

Namespace Description
varnish.cache_hit Number of requests served by the cache server
varnish.cache_miss Number of requests served by the backend server
varnish.uptime How long the child process has been running

Optional Metrics

Note

All stats that varnishstat reports are available in AppOptics. To expose these, you will need to edit the stats configuration variable in the /opt/appoptics/etc/plugins.d/varnish.yaml file. It is highly recommended to only expose metrics you are interested in as setting stats to "*" will cause all metrics (including those reporting no value/zero values) to start reporting to AppOptics at a 60 second resolution.

MAIN Metrics

Namespace Description
varnish.sess_conn Accepted client connections
varnish.sess_drop Dropped client connections
varnish.sess_fail Failed client connections
varnish.client_req_400 Client requests received, subject to 400 errors
varnish.client_req_417 Client requests received, subject to 417 errors
varnish.client_req Good client requests received
varnish.cache_hitpass Requests passed to the backend from the cache by choice
varnish.backend_conn Successful backend connections
varnish.backend_unhealthy Backend connections not attempted due to unhealthy backend state
varnish.backend_busy Unsuccessful backend connections due to exceeding the maximum number of connections
varnish.backend_fail Backend connection failures
varnish.backend_reuse Backend connection reuses
varnish.backend_recycle Backend connection recycles
varnish.backend_retry Backend connection retries
varnish.fetch_head Backend HEAD requests
varnish.fetch_length Backend responses with Content-Length in the body
varnish.fetch_chunked Backend responses that were chunked
varnish.fetch_eof Backend responses with EOF in the body
varnish.fetch_bad Backend responses with a bad body
varnish.fetch_none Backend responses with no body
varnish.fetch_1xx Backend responses with no body due to 1xx responses
varnish.fetch_204 Backend responses with no body due to 204 responses
varnish.fetch_304 Backend responses with no body due to 304 responses
varnish.fetch_failed Backend response fetches that failed
varnish.fetch_no_thread Backend response fetches that failed due to no thread being available
varnish.pools Number of thread pools
varnish.threads Total number of threads
varnish.threads_limited Number of threads that hit the thread pool limit
varnish.threads_created Number of threads created
varnish.threads_destroyed Number of threads destroyed
varnish.threads_failed Number of threads that failed to be created
varnish.thread_queue_len Length of session queue for threads
varnish.busy_sleep Number of requests sent to sleep because a busy object was found
varnish.busy_wakeup Number of requests woken after sleep on busy and rescheduled
varnish.busy_killed Number of requests killed after sleep on busy
varnish.sess_queued Client connections queued waiting for a thread
varnish.sess_dropped Client connections dropped waiting for a thread
varnish.n_object Number of object structs made
varnish.n_vampireobject Number of unresurrected objects
varnish.n_objectcore Number of objectcore structs made
varnish.n_objecthead Number of objecthead structs made
varnish.n_waitinglist Number of waitinglist structs made
varnish.n_backend Number of backends
varnish.n_expired Number of expired objects
varnish.n_lru_nuked Number of LRU nuked objects
varnish.n_lru_moved Number of LRU moved objects
varnish.losthdr HTTP header overflows
varnish.s_sess Total sessions seen
varnish.s_req Total requests seen
varnish.s_pipe Total pipe sessions seen
varnish.s_pass Total passed requests seen
varnish.s_fetch Total backend fetches initiated
varnish.s_synth Total synthethic responses made
varnish.s_req_hdrbytes Total request header bytes
varnish.s_req_bodybytes Total request body bytes
varnish.s_resp_hdrbytes Total response header bytes
varnish.s_resp_bodybytes Total response body bytes
varnish.s_pipe_hdrbytes Total pipe request header bytes
varnish.s_pipe_in Total piped bytes from client
varnish.s_pipe_out Total piped bytes to client
varnish.sess_closed Number of client connections closed
varnish.sess_closed_err Number of client connections closed with an error
varnish.sess_readahead Number of client connections read ahead
varnish.sess_herd Number of times the timeout_linger triggered
varnish.sc_rem_close Number of client connection closes with REM_CLOSE
varnish.sc_req_close Number of client connection closes with REQ_CLOSE
varnish.sc_req_http10 Number of client connection closes with error REQ_HTTP10
varnish.sc_rx_bad Number of client connection closes with error RX_BAD
varnish.sc_rx_body Number of client connection closes with error RX_BODY
varnish.sc_rx_junk Number of client connection closes with error RX_JUNK
varnish.sc_rx_overflow Number of client connection closes with error RX_OVERFLOW
varnish.sc_rx_timeout Number of client connection closes with error RX_TIMEOUT
varnish.sc_tx_pipe Number of client connection closes with TX_PIPE
varnish.sc_tx_error Number of client connection closes with error TX_ERROR
varnish.sc_tx_eof Number of client connection closes with TX_EOF
varnish.sc_resp_close Number of client connection closes with RESP_CLOSE
varnish.sc_overload Number of client connection closes with error OVERLOAD
varnish.sc_pipe_overflow Number of client connection closes with error PIPE_OVERFLOW
varnish.sc_range_short Number of client connection closes with error RANGE_SHORT
varnish.shm_records SHM records
varnish.shm_writes SHM writes
varnish.shm_flushes SHM flushes due to overflow
varnish.shm_cont SHM MTX contention
varnish.shm_cycles SHM cycles through buffer
varnish.backend_req Backend requests made
varnish.n_vcl Total number of loaded VCLs
varnish.n_vcl_avail Total number of VCLs available
varnish.n_vcl_discard Total number of discarded VCLs
varnish.bans Count of bans
varnish.bans_completed Number of bans marked ‘completed’ (no longer active)
varnish.bans_obj Number of bans using obj.*
varnish.bans_req Number of bans using req.*
varnish.bans_added Bans added to ban list
varnish.bans_deleted Bans deleted from ban list
varnish.bans_tested Bans tested against objects during lookup
varnish.bans_obj_killed Objects killed by bans during lookup
varnish.bans_lurker_tested Bans tested against objects by the ban-lurker
varnish.bans_tests_tested Ban tests tested against objects during lookup
varnish.bans_lurker_tests_tested Ban tests tested against objects by the ban-lurker
varnish.bans_lurker_obj_killed Objects killed by the ban-lurker
varnish.bans_dups Bans superseded by bans added later to the ban list
varnish.bans_lurker_contention Number of times the ban-lurker waited for lookups
varnish.bans_persisted_bytes Bytes used by the persisted ban lists
varnish.bans_persisted_fragmentation Extra bytes in persisted ban lists due to fragmentation
varnish.n_purges Number of purge operations executed
varnish.n_obj_purged Number of purged objects
varnish.exp_mailed Number of objects mailed to expiry thread
varnish.exp_received Number of objects received by expiry thread
varnish.hcb_nolock HCB Lookups without lock
varnish.hcb_lock HCB Lookups with lock
varnish.hcb_insert HCB Inserts
varnish.esi_errors Edge Side Includes (ESI) parse errors
varnish.esi_warnings Edge Side Includes (ESI) parse warnings (unlock)
varnish.vmods Loaded VMODs
varnish.n_gzip Number of Gzip operations
varnish.n_gunzip Number of Gunzip operations
varnish.vsm_free Free VSM space
varnish.vsm_used Used VSM space
varnish.vsm_cooling Soon to be freed VSM space
varnish.vsm_overflow Data which does not fit in the VSM space
varnish.vsm_overflowed Total data which did not fit in the VSM space

MGT Metrics

Namespace Description
varnish.child_start Child processes started
varnish.child_exit Child processes that were stopped normally
varnish.child_stop Child processes that exited with an unexpected return code
varnish.child_died Child processes that died due to signals
varnish.child_dump Child processes that produced core dumps
varnish.child_panic Child processes that panicked

MEMPOOL Metrics

Namespace Description
varnish.busyobj.live In use
varnish.busyobj.pool In Pool
varnish.busyobj.sz_wanted Size requested
varnish.busyobj.sz_actual Size allocated
varnish.busyobj.allocs Allocations
varnish.busyobj.frees Frees
varnish.busyobj.recycle Recycled from pool
varnish.busyobj.timeout Timed out from pool
varnish.busyobj.toosmall Too small to recycle
varnish.busyobj.surplus Too many for pool
varnish.busyobj.randry Pool ran dry
varnish.req0.live In use
varnish.req0.pool In Pool
varnish.req0.sz_wanted Size requested
varnish.req0.sz_actual Size allocated
varnish.req0.allocs Allocations
varnish.req0.frees Frees
varnish.req0.recycle Recycled from pool
varnish.req0.timeout Timed out from pool
varnish.req0.toosmall Too small to recycle
varnish.req0.surplus Too many for pool
varnish.req0.randry Pool ran dry
varnish.sess0.live In use
varnish.sess0.pool In Pool
varnish.sess0.sz_wanted Size requested
varnish.sess0.sz_actual Size allocated
varnish.sess0.allocs Allocations
varnish.sess0.frees Frees
varnish.sess0.recycle Recycled from pool
varnish.sess0.timeout Timed out from pool
varnish.sess0.toosmall Too small to recycle
varnish.sess0.surplus Too many for pool
varnish.sess0.randry Pool ran dry
varnish.req1.live In use
varnish.req1.pool In Pool
varnish.req1.sz_wanted Size requested
varnish.req1.sz_actual Size allocated
varnish.req1.allocs Allocations
varnish.req1.frees Frees
varnish.req1.recycle Recycled from pool
varnish.req1.timeout Timed out from pool
varnish.req1.toosmall Too small to recycle
varnish.req1.surplus Too many for pool
varnish.req1.randry Pool ran dry
varnish.sess1.live In use
varnish.sess1.pool In Pool
varnish.sess1.sz_wanted Size requested
varnish.sess1.sz_actual Size allocated
varnish.sess1.allocs Allocations
varnish.sess1.frees Frees
varnish.sess1.recycle Recycled from pool
varnish.sess1.timeout Timed out from pool
varnish.sess1.toosmall Too small to recycle
varnish.sess1.surplus Too many for pool
varnish.sess1.randry Pool ran dry

SMA Metrics

Namespace Description
varnish.s0.c_req Allocator requests
varnish.s0.c_fail Allocator failures
varnish.s0.c_bytes Bytes allocated
varnish.s0.c_freed Bytes freed
varnish.s0.g_alloc Allocations outstanding
varnish.s0.g_bytes Bytes outstanding
varnish.s0.g_space Bytes available
varnish.Transient.c_req Allocator requests
varnish.Transient.c_fail Allocator failures
varnish.Transient.c_bytes Bytes allocated
varnish.Transient.c_freed Bytes freed
varnish.Transient.g_alloc Allocations outstanding
varnish.Transient.g_bytes Bytes outstanding
varnish.Transient.g_space Bytes available

VBE Metrics

Namespace Description
varnish.boot.default.happy Number of happy health probes
varnish.boot.default.bereq_hdrbytes Total request header bytes
varnish.boot.default.bereq_bodybytes Total request body bytes
varnish.boot.default.beresp_hdrbytes Total response header bytes
varnish.boot.default.beresp_bodybytes Total response body bytes
varnish.boot.default.pipe_hdrbytes Total pipe request header bytes
varnish.boot.default.pipe_out Total piped bytes to backend
varnish.boot.default.pipe_in Total piped bytes from backend
varnish.boot.default.conn Number of concurrent connections to backend
varnish.boot.default.req Number of backend requests sent

LCK Metrics

Namespace Description
varnish.backend.creat Created locks
varnish.backend.destroy Destroyed locks
varnish.backend.locks Lock Operations
varnish.backend_tcp.creat Created locks
varnish.backend_tcp.destroy Destroyed locks
varnish.backend_tcp.locks Lock Operations
varnish.ban.creat Created locks
varnish.ban.destroy Destroyed locks
varnish.ban.locks Lock Operations
varnish.busyobj.creat Created locks
varnish.busyobj.destroy Destroyed locks
varnish.busyobj.locks Lock Operations
varnish.cli.creat Created locks
varnish.cli.destroy Destroyed locks
varnish.cli.locks Lock Operations
varnish.exp.creat Created locks
varnish.exp.destroy Destroyed locks
varnish.exp.locks Lock Operations
varnish.hcb.creat Created locks
varnish.hcb.destroy Destroyed locks
varnish.hcb.locks Lock Operations
varnish.lru.creat Created locks
varnish.lru.destroy Destroyed locks
varnish.lru.locks Lock Operations
varnish.mempool.creat Created locks
varnish.mempool.destroy Destroyed locks
varnish.mempool.locks Lock Operations
varnish.objhdr.creat Created locks
varnish.objhdr.destroy Destroyed locks
varnish.objhdr.locks Lock Operations
varnish.pipestat.creat Created locks
varnish.pipestat.destroy Destroyed locks
varnish.pipestat.locks Lock Operations
varnish.sess.creat Created locks
varnish.sess.destroy Destroyed locks
varnish.sess.locks Lock Operations
varnish.smp.creat Created locks
varnish.smp.destroy Destroyed locks
varnish.smp.locks Lock Operations
varnish.vbe.creat Created locks
varnish.vbe.destroy Destroyed locks
varnish.vbe.locks Lock Operations
varnish.vcapace.creat Created locks
varnish.vcapace.destroy Destroyed locks
varnish.vcapace.locks Lock Operations
varnish.vcl.creat Created locks
varnish.vcl.destroy Destroyed locks
varnish.vcl.locks Lock Operations
varnish.vxid.creat Created locks
varnish.vxid.destroy Destroyed locks
varnish.vxid.locks Lock Operations
varnish.waiter.creat Created locks
varnish.waiter.destroy Destroyed locks
varnish.waiter.locks Lock Operations
varnish.wq.creat Created locks
varnish.wq.destroy Destroyed locks
varnish.wq.locks Lock Operations
varnish.wstat.creat Created locks
varnish.wstat.destroy Destroyed locks
varnish.wstat.locks Lock Operations
varnish.sma.creat Created locks
varnish.sma.destroy Destroyed locks
varnish.sma.locks Lock Operations

Tags

The table below outlines the default set of tags provided for each metric.

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias.
section Prefix of the original varnish stat (one of main, mgt, mempool, sma, vbe, lck)