Sensu is a commonly used monitoring and telemetry application comprising servers, clients, a database and a message bus. This article outlines how to configure Cumulus Linux to operate as a Sensu client. I also offers some tips on installing and configuring a set of Sensu related servers (server, api, redis, rabbitmq, influxdb).
You can read more about monitoring Cumulus Linux with Sensu.
{{table_of_contents}}
Components
As outlined in the above link, Sensu has multiple components. This simplified guide outlines how to install these components and get Sensu server working with a Cumulus Linux Sensu Client.
The components covered in this guide include:
- Sensu client running on Cumulus Linux
- RabbitMQ server message broker.
- Sensu server, which receives data from RabbitMQ and writes that data to a secure location
- Redis as a database that stores the data received by the Sensu server
- Sensu API and Uchiwa server as a web front end GUI (often called a Dashboard)
In practice, each component can be deployed on a different computer but for the sake of simplicity, RabbitMQ, Sensu Server, Redis, Sensu API and Uchiwa will all be configured on the same workstation and only Sensu Client will be running on Cumulus Linux.
Installing Sensu Packages
In order to get the Cumulus Linux Sensu Client on the switch to communicate with a Sensu server (via RabbitMQ), install the latest Sensu package directly from the Sensu repo, since the Cumulus Linux and Debian Jessie repositories do not currently have the Sensu meta package:
# sensu install on switch wget -q http://repositories.sensuapp.org/apt/pubkey.gpg -O- | apt-key add - echo "deb http://repositories.sensuapp.org/apt sensu main" | tee /etc/apt/sources.list.d/sensu.list apt-get update apt-get install sensu # install sensu (Ruby) checks we will be called on to run sensu-install -p process-checks sensu-install -p cpu-checks
sensu-install -p load-checks
Note that other check scripts (whether RubyGems or those written in other languages) may need to be installed depending on what needs to be monitored.
Configuring the Sensu Client
On the Cumulus Linux switch, create the file /etc/sensu/config.json. Here, the RabbitMQ configuration gives us access to the message broker (RabbitMQ) while the client configuration simply identifies the switch and sets the subscription type for responding to requests from the server. (Note: All usernames and passwords should be changed. The usernames and passwords shown here are for testing and are not for production use.)
{
"rabbitmq": {
"host": "10.1.1.50",
"port": 5672,
"user": "sensu",
"password": "password",
"vhost": "/sensu"
},
"client": {
"name": "CumulusLinux",
"address": "localhost",
"subscriptions": ["cumulus"],
"safe_mode": false,
"keepalive": { }
}
}
Once the client is configured properly, subsequent fine tuning and data-gathering can be configured on the Sensu Server.
After configuring, use systemctl to enable, start and check status on the sensu-client:
systemctl status sensu-client systemctl enable sensu-client systemctl restart sensu-client
# This shows startup errors to make sure the client was started correctly systemctl status sensu-client # check the logs for errors or warnings as you start or restart the sensu-client tail -f /var/log/sensu/sensu-client.log
Server Component Configuration
On a separate Debian Jessie workstation, install RabbitMQ, Sensu, Redis and Uchiwa.
In this article, a Debian Jessie server is used to install and configure RabbitMQ, Sensu Server, Sensu API, Redis and the Uchiwa web server.
Installing Sensu Server, Redis and Uchiwa
Since Sensu is not part of the default set of Jessie and Cumulus Linux packages, as stated earlier, add the Sensu repo to apt, then install Sensu, Redis and Uchiwa:
# Add the public Sensu key wget -q http://repositories.sensuapp.org/apt/pubkey.gpg -O- | apt-key add - echo "deb http://repositories.sensuapp.org/apt sensu main"| tee /etc/apt/sources.list.d/sensu.list apt-get update # install it along with an uchiwa dashboard (optional) and a redis server apt-get install sensu apt-get install uchiwa
apt-get install redis-server
Installing and Configuring RabbitMQ
RabbitMQ serves as the message broker and queues up requests and responses from the clients and the Sensu server. RabbitMQ is included in the default Debian Jessie repo; installing RabbitMQ also installs the required Erlang Debian packages (RabbitMQ is written in the Erlang language). Again, remember to change these usernames and passwords as they are only suitable for testing and are not for production use.
# install rabbitmq-server apt-get install rabbitmq-server
Configuring RabbitMQ
Configure the message broker with a file called /etc/rabbitmq/rabbitmq.config.
Note: This is an Erlang config file, not JSON.
[
{rabbit, [
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true,
{backlog, 128,
{nodelay, true,
{exit_on_close, false}]
},
{default_user, <<"sensu">>},
{default_pass, <<"password">>}
]},
{kernel, [
]}
,
{rabbitmq_management, [
{listener, [
{port, 15672}
]}
]}
].
% EOF
Also create an admin configuration file called /etc/rabbitmq/rabbitmqadmin.conf to monitor RabbitMQ with a browser on this port (http:<server IP>:15672).
[default] port = 15672
Starting the RabbitMQ Message Broker and Adding a User
/etc/init.d/rabbitmq-server start # create a user/password for rabbitmq. Certs can also be generated and used rabbitmqctl add_vhost /sensu rabbitmqctl add_user sensu password rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*" rabbitmqctl status # this will start a management server we can connect to with a browser. rabbitmq-plugins enable rabbitmq_management
Configuring the Sensu API
The Sensu API is needed for access via Uchiwa (or some other dashboard browser) this reads from Redis. The following config should be placed in /etc/sensu/conf.d/api.json:
{
"api": {
"bind": "0.0.0.0",
"port": 4567,
"host": "127.0.0.1"
}
}
Configuring the Sensu Server
The Sensu server needs to connect to the RabbitMQ message broker and initiate check requests to clients. The Sensu server then takes this information and writes it to Redis. The following config should be placed in /etc/sensu/conf.d/rabbitmq.json:
{
"rabbitmq": {
"port": 5672,
"host": "127.0.0.1",
"user": "sensu",
"password": "password",
"vhost": "/sensu",
"reconnect_on_error": false
}
}
Configuring Server Checks
If you need to configure checks and metric collection scripts for the Sensu server that run on subscribed clients, the following check configurations can be placed in /etc/sensu/conf.d/checks/. This check, called check_cpu.json, checks the cpu on the client and returns "WARNING" if the CPU is above 70% and "CRITICAL" if the CPU is above 90%. Otherwise, it returns "OK".
{
"checks": {
"cpu": {
"command": "check-cpu.rb -w 70 -c 90",
"interval": 60,
"subscribers": ["cumulus"]
}
}
}
This checks for a process called "cron" and can be saved in a file named /etc/sensu/conf.d/checks/check_cron.json
{
"checks": {
"cron": {
"command": "check-process.rb -p cron",
"subscribers": ["cumulus"],
"interval": 60
}
}
This next check returns CPU metrics and is named /etc/sensu/conf.d/checks/check_process.json
{
"checks": {
"process": {
"type": "metric",
"command": "metrics-cpu.rb",
"subscribers": ["cumulus"],
"interval": 10
}
}
}
Configuring Redis Connectivity
Redis is responsible for storing the results. This configuration lets the Sensu server store data in the Redis database. The following configuration (for sensu) should be saved in /etc/sensu/conf.d/redis.json:
{
"redis": {
"port": 6379,
"host": "127.0.0.1",
"reconnect_on_error": true
}
}
Configuring the Uchiwa Web Server
The Uchiwa web server communicates with sensu-api server and displays a simple dashboard to show Sensu status. Its purpose is to monitor the server events and client status. The sensu part of the Uchiwa server config should be saved in /etc/sensu/uchiwa.json.
{
"sensu": [
{
"name": "sensu",
"host": "127.0.0.1",
"ssl": false,
"insecure": false,
"port": 4567,
"user": "sensu",
"pass": "sensu",
"path": "",
"timeout": 5
}
],
"uchiwa": {
"host": "0.0.0.0",
"port": 3000,
"user": "sensu",
"pass": "sensu",
"refresh": 5
}
}
Restarting All Services
Restart all Sensu-related services to enable the configuration.
# the rabbitMQ server was probably started in one of the previous steps systemctl restart rabbitmq-server # we use the default redis configs and simple send data from the sensu server systemctl restart redis-server # this is the actual server that is sending requests (check, metric, etc) to the clients systemctl restart sensu-server # this is needed to allow uchiwa to monitor the events and checks systemctl restart sensu-api # start up a simple monitoring dashboard systemctl restart uchiwa
Additional Databases and Visualization
Using time-series tables in databases can simplify visualization of various metrics that can be gathered automatically from the Cumulus Linux switches and routers.
The first step is to install a small package called InfluxDB which uses a simple sqlite3 database as a default. InfluxDB can be configured to use a different database but for simplicity, the default sqlite DB will be used here. Installation
# install stable influxDB and chronograf
wget https://dl.influxdata.com/influxdb/releases/influxdb_0.12.2-1_amd64.deb
# chronograf is a useful alternative to grafana
wget https://dl.influxdata.com/chronograf/releases/chronograf_0.13.0_amd64.deb
# install the packages
dpkg -i influxdb_0.12.2-1_amd64.deb
dpkg -i chronograf_0.13.0_amd64.deb
systemctl enable influxdb
systemctl start influxdb
Now, create a database with a user and password (Note: make sure to change the username and password since those used here are not secure).
# Use the commandline influx command to create sensu DB and a user
influx
> CREATE DATABASE sensu
> CREATE USER "sensu" WITH PASSWORD 'sensu' WITH ALL PRIVILEGES
Now, edit the config file /etc/influxdb/influxdb.conf for influxDB and make sure the ports are enabled correctly
# make sure port 8086 is used for http for sensu
[http]
enabled = true
bind-address = ":8086"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/influxdb.pem"
max-row-limit = 10000
# add udp access for
[[udp]]
enabled = true
bind-address = ":8090"
database = "sensu"
# restart influxdb
systemctl restart influxdb
On the influxdb server, make sure to install em-http-request
/opt/sensu/embedded/bin/gem install em-http-request
And put the ruby handler file from https://github.com/seegno/sensu-influxdb-extension/influxdb.rb in /etc/sensu/extensions/influxdb.rb and chmod 777
Then add influxdb as a handler option in /etc/sensu/conf.d/metrics.json
{
"handlers": {
"metrics": {
"type": "set",
"handlers": [ "debug", "influxdb"]
}
}
}
Now create a influxdb handler in /etc/sensu/conf.d/influxdb.json
{
"influxdb": {
"host": "localhost",
"port": "8086",
"user": "sensu",
"password": "sensu",
"ssl_enable": false,
"database": "sensu"
}
}
These metrics (load, cpu) need to be installed on switch (sensu Client) (metrics-load.rm, metrics-cpu.rb) with the command sensu-install load-checks and make sure that the metric check uses it in the config /etc/sensu/conf.d/checks/metrics-load.json. This metric gathering command will be run on all "cumulus" subscribers every 60 seconds and the results will be passed to the metrics handler. This metrics handler will run the influxdb Ruby script on the output and then the result will be fed into the InfluxDB database.
{
"checks": {
"metrics-load": {
"type": "metric",
"command": "metrics-load.rb",
"subscribers": ["cumulus","basics"],
"handlers": ["metrics"],
"interval": 10,
"time_precision": "s",
"influxdb": {
"database": "sensu",
"tags": {
"stage": "cumulus",
"region": "eu-west-1"
}
}
}
}
}
The data in the InfluxDB tables can be checked like any other SQL database.
# interactive DB
influx
> show databases
# select the sensu database
> use sensu
> show measurements
cel-redxp-01.load_avg.fifteen
cel-redxp-01.load_avg.five
cel-redxp-01.load_avg.one
> select * from "cel-redxp-01.load_avg.one"
name: cel-redxp-01.load_avg.one
-------------------------------
time duration host region stage value
1463166067000000000 0.364 CumulusLinux-3.0.0 eu-west-1 cumulus 0.88
1463166077000000000 0.364 CumulusLinux-3.0.0 eu-west-1 cumulus 0.89
Visualization
Visualization of the data being gathered in this InfluxDB can be easily accomplished with Chronograf or Grafana.
Troubleshooting
To check whether the configuration was correctly read for the various servers and the client, use systemctl status to see any errors during startup. For example, if the sensu client configuration had an extra comma, you would see the following as you start the client:
root@cel-redxp-01:/opt# systemctl start sensu-client
Job for sensu-client.service failed. See 'systemctl status sensu-client.service' and 'journalctl -xn' for details.
root@cel-redxp-01:/opt#
root@cel-redxp-01:/opt# systemctl status sensu-client
\u25cf sensu-client.service - LSB: Sensu monitoring framework client
Loaded: loaded (/etc/init.d/sensu-client)
Active: failed (Result: exit-code) since Sun 2016-05-15 16:54:08 UTC; 17s ago
Process: 7536 ExecStart=/etc/init.d/sensu-client start (code=exited, status=1/FAILURE)
May 15 16:54:08 cel-redxp-01 sensu-client[7536]: Starting sensu-client: failed!
May 15 16:54:08 cel-redxp-01 systemd[1]: sensu-client.service: control process exited, code=exited status=1
May 15 16:54:08 cel-redxp-01 systemd[1]: Failed to start LSB: Sensu monitoring framework client.
May 15 16:54:08 cel-redxp-01 systemd[1]: Unit sensu-client.service entered failed state.
root@cel-redxp-01:/opt# tail /var/log/sensu/sensu-client.log
{"timestamp":"2016-05-15T16:54:07.692837+0000","level":"warn","message":"loading config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-05-15T16:54:07.693062+0000","level":"warn","message":"config file must be valid json","file":"/etc/sensu/config.json","error":"expected hash key, not a hash close at line 8, column 4 [parse.c:547]"}
{"timestamp":"2016-05-15T16:54:07.693211+0000","level":"warn","message":"ignoring config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-05-15T16:54:07.693334+0000","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-05-15T16:54:07.693572+0000","level":"fatal","message":"invalid settings"}
{"timestamp":"2016-05-15T16:54:07.693698+0000","level":"fatal","message":"client must be a hash","object":null}
{"timestamp":"2016-05-15T16:54:07.693798+0000","level":"fatal","message":"SENSU NOT RUNNING!"}
This last log snippet from /var/log/sensu/sensu-client.log shows that the configuration was incorrect and the client did not start.
Comments