Sensu is a commonly used monitoring and telemetry application comprising servers, clients, a database and a message bus. This article outlines how to configure Cumulus Linux to operate as a Sensu client.   I also offers some tips on installing and configuring a set of Sensu related servers (server, api, redis, rabbitmq, influxdb).

You can read more about monitoring Cumulus Linux with Sensu.

Components

As outlined in the above link, Sensu has multiple components. This simplified guide outlines how to install these components and get Sensu server working with a Cumulus Linux Sensu Client.

The components covered in this guide include:

• Sensu client running on Cumulus Linux
• RabbitMQ server message broker.
• Sensu server, which receives data from RabbitMQ and writes that data to a secure location
• Redis as a database that stores the data received by the Sensu server
• Sensu API and Uchiwa server as a web front end GUI (often called a Dashboard)

In practice, each component can be deployed on a different computer but for the sake of simplicity, RabbitMQ, Sensu Server, Redis, Sensu API and Uchiwa will all be configured on the same workstation and only Sensu Client will be running on Cumulus Linux.

Installing Sensu Packages

In order to get the Cumulus Linux Sensu Client on the switch to communicate with a Sensu server (via RabbitMQ), install the latest Sensu package directly from the Sensu repo, since the Cumulus Linux and Debian Jessie repositories do not currently have the Sensu meta package:

# sensu install on switch 
wget -q http://repositories.sensuapp.org/apt/pubkey.gpg -O- | apt-key add -
echo "deb http://repositories.sensuapp.org/apt sensu main" | tee /etc/apt/sources.list.d/sensu.list
apt-get update
apt-get install sensu
# install sensu (Ruby) checks we will be called on to run
sensu-install -p process-checks
sensu-install -p cpu-checks
sensu-install -p load-checks

Note that other check scripts (whether RubyGems or those written in other languages) may need to be installed depending on what needs to be monitored.

Configuring the Sensu Client

On the Cumulus Linux switch, create the file /etc/sensu/config.json. Here, the RabbitMQ configuration gives us access to the message broker (RabbitMQ) while the client configuration simply identifies the switch and sets the subscription type for responding to requests from the server. (Note: All usernames and passwords should be changed.  The usernames and passwords shown here are for testing and are not for production use.)

{
  "rabbitmq": {
    "host": "10.1.1.50",
    "port": 5672,
    "user": "sensu",
    "password": "password",
    "vhost": "/sensu"
  },
  "client": {
    "name": "CumulusLinux",
    "address": "localhost",
    "subscriptions": ["cumulus"],
    "safe_mode": false,
    "keepalive": { }
  }
}

Once the client is configured properly, subsequent fine tuning and data-gathering can be configured on the Sensu Server.   

After configuring, use systemctl to enable, start and check status on the sensu-client:

systemctl status sensu-client
systemctl enable sensu-client
systemctl restart sensu-client
# This shows startup errors to make sure the client was started correctly
systemctl status sensu-client
# check the logs for errors or warnings as you start or restart the sensu-client
tail -f /var/log/sensu/sensu-client.log

Server Component Configuration

On a separate Debian Jessie workstation, install RabbitMQ, Sensu, Redis and Uchiwa.

In this article, a Debian Jessie server is used to install and configure RabbitMQ, Sensu Server, Sensu API, Redis and the Uchiwa web server.

Installing Sensu Server, Redis and Uchiwa

Since Sensu is not part of the default set of Jessie and Cumulus Linux packages, as stated earlier, add the Sensu repo to apt, then install Sensu, Redis and Uchiwa:

# Add the public Sensu key
wget -q   http://repositories.sensuapp.org/apt/pubkey.gpg -O- | apt-key add -
echo "deb http://repositories.sensuapp.org/apt sensu main"| tee /etc/apt/sources.list.d/sensu.list
apt-get update
# install it along with an uchiwa dashboard (optional) and a redis server
apt-get install sensu
apt-get install uchiwa 
apt-get install redis-server

Installing and Configuring RabbitMQ

RabbitMQ serves as the message broker and queues up requests and responses from the clients and the Sensu server. RabbitMQ is included in the default Debian Jessie repo; installing RabbitMQ also installs the required Erlang Debian packages (RabbitMQ is written in the Erlang language).   Again, remember to change these usernames and passwords as they are only suitable for testing and are not for production use.

# install rabbitmq-server
apt-get install rabbitmq-server

Configuring RabbitMQ

Configure the message broker with a file called /etc/rabbitmq/rabbitmq.config.

Note: This is an Erlang config file, not JSON.

[
  {rabbit, [
    {tcp_listen_options,
         [binary,
         {packet,        raw},
         {reuseaddr,     true,
         {backlog,       128,
         {nodelay,       true,
         {exit_on_close, false}]
    },
    {default_user, <<"sensu">>},
    {default_pass, <<"password">>}
  ]},
  {kernel, [

  ]}
,
  {rabbitmq_management, [
    {listener, [
      {port, 15672}
    ]}
  ]}
].
% EOF

Also create an admin configuration file called /etc/rabbitmq/rabbitmqadmin.conf to monitor RabbitMQ with a browser on this port (http:<server IP>:15672).

[default]
port = 15672

Starting the RabbitMQ Message Broker and Adding a User

/etc/init.d/rabbitmq-server start
# create a user/password for rabbitmq.  Certs can also be generated and used
rabbitmqctl add_vhost /sensu
rabbitmqctl add_user sensu password
rabbitmqctl set_user_tags sensu administrator
rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"
rabbitmqctl status
# this will start a management server we can connect to with a browser.
rabbitmq-plugins enable rabbitmq_management

Configuring the Sensu API

The Sensu API is needed for access via Uchiwa (or some other dashboard browser) this reads from Redis.  The following config should be placed in /etc/sensu/conf.d/api.json:

{
  "api": {
    "bind": "0.0.0.0",
    "port": 4567,
    "host": "127.0.0.1"
  }
}

Configuring the Sensu Server

The Sensu server needs to connect to the RabbitMQ message broker and initiate check requests to clients. The Sensu server then takes this information and writes it to Redis. The following config should be placed in /etc/sensu/conf.d/rabbitmq.json:

{
  "rabbitmq": {
    "port": 5672,
    "host": "127.0.0.1",
    "user": "sensu",
    "password": "password",
    "vhost": "/sensu",
    "reconnect_on_error": false
  }
}

Configuring Server Checks

If you need to configure checks and metric collection scripts for the Sensu server that run on subscribed clients, the following check configurations can be placed in /etc/sensu/conf.d/checks/.   This check, called check_cpu.json, checks the cpu on the client and returns "WARNING" if the CPU is above 70% and "CRITICAL" if the CPU is above 90%.  Otherwise, it returns "OK".

{
  "checks": {
    "cpu": {
      "command": "check-cpu.rb -w 70 -c 90",
      "interval": 60,
      "subscribers": ["cumulus"]
    }
  }
}


This checks for a process called "cron" and can be saved in a file named  /etc/sensu/conf.d/checks/check_cron.json

{
  "checks": {
    "cron": {
      "command": "check-process.rb -p cron",
      "subscribers": ["cumulus"],
      "interval": 60
    }
  }

This next check returns CPU metrics and is named /etc/sensu/conf.d/checks/check_process.json

{
  "checks": {
    "process": {
      "type": "metric",
      "command": "metrics-cpu.rb",
      "subscribers": ["cumulus"],
      "interval": 10
    }
  }
}

Configuring Redis Connectivity

Redis is responsible for storing the results. This configuration lets the Sensu server store data in the Redis database.  The following configuration (for sensu) should be saved in /etc/sensu/conf.d/redis.json:

{
  "redis": {
  "port": 6379,
  "host": "127.0.0.1",
  "reconnect_on_error": true
  }
}

Configuring the Uchiwa Web Server

The Uchiwa web server communicates with sensu-api server and displays a simple dashboard to show Sensu status. Its purpose is to monitor the server events and client status.   The sensu part of the Uchiwa server config should be saved in /etc/sensu/uchiwa.json.

{
  "sensu": [
    {
      "name": "sensu",
      "host": "127.0.0.1",
      "ssl": false,
      "insecure": false,
      "port": 4567,
      "user": "sensu",
      "pass": "sensu",
      "path": "",
      "timeout": 5
    }
  ],
  "uchiwa": {
    "host": "0.0.0.0",
    "port": 3000,
    "user": "sensu",
    "pass": "sensu",
    "refresh": 5
  }
}

Restarting All Services

Restart all Sensu-related services to enable the configuration.

# the rabbitMQ server was probably started in one of the previous steps
systemctl restart rabbitmq-server
# we use the default redis configs and simple send data from the sensu server
systemctl restart redis-server
# this is the actual server that is sending requests (check, metric, etc) to the clients
systemctl restart sensu-server
# this is needed to allow uchiwa to monitor the events and checks
systemctl restart sensu-api
# start up a simple monitoring dashboard
systemctl restart uchiwa

Additional Databases and Visualization

Using time-series tables in databases can simplify visualization of various metrics that can be gathered automatically from the Cumulus Linux switches and routers.

The first step is to install a small package called InfluxDB which uses a simple sqlite3 database as a default.  InfluxDB can be configured to use a different database but for simplicity, the default sqlite DB will be used here.   Installation 

# install stable influxDB and chronograf
wget https://dl.influxdata.com/influxdb/releases/influxdb_0.12.2-1_amd64.deb
# chronograf is a useful alternative to grafana
wget https://dl.influxdata.com/chronograf/releases/chronograf_0.13.0_amd64.deb
# install the packages
dpkg -i influxdb_0.12.2-1_amd64.deb
dpkg -i chronograf_0.13.0_amd64.deb
systemctl enable influxdb
systemctl start influxdb

Now, create a database with a user and password (Note: make sure to change the username and password since those used here are not secure).

# Use the commandline influx command to create sensu DB and a user
influx
> CREATE DATABASE sensu
> CREATE USER "sensu" WITH PASSWORD 'sensu' WITH ALL PRIVILEGES



Now, edit the config file /etc/influxdb/influxdb.conf for influxDB and make sure the ports are enabled correctly

# make sure port 8086 is used for http for sensu
[http]
 enabled = true
 bind-address = ":8086"
 auth-enabled = false
 log-enabled = true
 write-tracing = false
 pprof-enabled = false
 https-enabled = false
 https-certificate = "/etc/ssl/influxdb.pem"
 max-row-limit = 10000

# add udp access for 
[[udp]]
 enabled = true
 bind-address = ":8090"
 database = "sensu"

# restart influxdb
systemctl restart influxdb

On the influxdb server, make sure to install em-http-request

/opt/sensu/embedded/bin/gem install em-http-request

And put the ruby handler file from https://github.com/seegno/sensu-influxdb-extension/influxdb.rb in /etc/sensu/extensions/influxdb.rb  and chmod 777

Then add influxdb as a handler option in /etc/sensu/conf.d/metrics.json

{
 "handlers": {
   "metrics": {
     "type": "set",
     "handlers": [ "debug", "influxdb"]
   }
  }
}

Now create a influxdb handler in /etc/sensu/conf.d/influxdb.json

{
 "influxdb": {
   "host": "localhost",
   "port": "8086",
   "user": "sensu",
   "password": "sensu",
   "ssl_enable": false,
   "database": "sensu"
 }
}

These metrics (load, cpu) need to be installed on switch (sensu Client)  (metrics-load.rm, metrics-cpu.rb) with the command  sensu-install load-checks and make sure that the metric check uses it in the config /etc/sensu/conf.d/checks/metrics-load.json.   This metric gathering command will be run on all "cumulus" subscribers every 60 seconds and the results will be passed to the metrics handler.   This metrics handler will run the influxdb Ruby script on the output and then the result will be fed into the InfluxDB database. 

{
  "checks": {
    "metrics-load": {
       "type": "metric",
       "command": "metrics-load.rb",
       "subscribers": ["cumulus","basics"],
       "handlers": ["metrics"],
       "interval": 10,
       "time_precision": "s",
       "influxdb": {
          "database": "sensu",
          "tags": {
             "stage": "cumulus",
             "region": "eu-west-1"
          }
       }
    }
  }
}


The data in the InfluxDB tables can be checked like any other SQL database.  


# interactive DB
influx
> show databases
# select the sensu database
> use sensu

> show measurements
cel-redxp-01.load_avg.fifteen
cel-redxp-01.load_avg.five
cel-redxp-01.load_avg.one

> select * from "cel-redxp-01.load_avg.one"
name: cel-redxp-01.load_avg.one
-------------------------------
time duration host region stage value
1463166067000000000 0.364 CumulusLinux-3.0.0 eu-west-1 cumulus 0.88
1463166077000000000 0.364 CumulusLinux-3.0.0 eu-west-1 cumulus 0.89

Visualization

Visualization of the data being gathered in this InfluxDB can be easily accomplished with Chronograf or Grafana.   

Troubleshooting

To check whether the configuration was correctly read for the various servers and the client, use systemctl status to see any errors during startup.   For example, if the sensu client configuration had an extra comma, you would see the following as you start the client:

root@cel-redxp-01:/opt# systemctl start sensu-client
Job for sensu-client.service failed. See 'systemctl status sensu-client.service' and 'journalctl -xn' for details.
root@cel-redxp-01:/opt# 
root@cel-redxp-01:/opt# systemctl status sensu-client
\u25cf sensu-client.service - LSB: Sensu monitoring framework client
 Loaded: loaded (/etc/init.d/sensu-client)
 Active: failed (Result: exit-code) since Sun 2016-05-15 16:54:08 UTC; 17s ago
 Process: 7536 ExecStart=/etc/init.d/sensu-client start (code=exited, status=1/FAILURE)

May 15 16:54:08 cel-redxp-01 sensu-client[7536]: Starting sensu-client: failed!
May 15 16:54:08 cel-redxp-01 systemd[1]: sensu-client.service: control process exited, code=exited status=1
May 15 16:54:08 cel-redxp-01 systemd[1]: Failed to start LSB: Sensu monitoring framework client.
May 15 16:54:08 cel-redxp-01 systemd[1]: Unit sensu-client.service entered failed state.

root@cel-redxp-01:/opt# tail /var/log/sensu/sensu-client.log
{"timestamp":"2016-05-15T16:54:07.692837+0000","level":"warn","message":"loading config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-05-15T16:54:07.693062+0000","level":"warn","message":"config file must be valid json","file":"/etc/sensu/config.json","error":"expected hash key, not a hash close at line 8, column 4 [parse.c:547]"}
{"timestamp":"2016-05-15T16:54:07.693211+0000","level":"warn","message":"ignoring config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-05-15T16:54:07.693334+0000","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-05-15T16:54:07.693572+0000","level":"fatal","message":"invalid settings"}
{"timestamp":"2016-05-15T16:54:07.693698+0000","level":"fatal","message":"client must be a hash","object":null}
{"timestamp":"2016-05-15T16:54:07.693798+0000","level":"fatal","message":"SENSU NOT RUNNING!"}

This last log snippet from /var/log/sensu/sensu-client.log shows that the configuration was incorrect and the client did not start.