This site uses cookies for analytics. By continuing to browse this site, you agree to use this.  Learn more

Setting up a monitoring server in minutes

Simplifying things with our new awesome cdist types.

Posted on June 4, 2017

An article by Kamila, our site reliability engineer.

Quickly set up a monitoring server

cdist is an interesting take on configuration management developed at ungleich. It is simple, versatile, and has no dependencies on the target system. In fact, it is a bit too simple compared to other, far more sophisticated configuration management solutions -- and that's precisely why I like it.

I use it for configuring all my monitoring servers (among other things). Below you can find a cdist manifest snippet that comes from our real configuration at DataCenterLight.

I have created several cdist types useful for installing a Prometheus monitoring server, as well as many exporters:

Installing golang and go packages

  • __golang_from_vendor: Installs golang from https://golang.org. This makes it easy to install any version of golang, including those that your old and obsolete distro doesn't have.

  • __go_get: Easy installation of golang packages straight from online repos. Useful for installing Prometheus itself or the various exporters.

Daemontools

Daemontools are awesome. In line with the Unix philosophy, they do one thing and do it well. In this case they make sure that services run. Without the messy parts!

  • __daemontools: Installs and configures Prometheus server. Yay!
     
  • __daemontools_service: Creates a service directory. I gladly use this for everything that doesn't come with a good init script, such as Prometheus and exporters. It's so simple!

High-level types for Prometheus and friends

Note: these are not merged in upstream yet. I will update with links when they are!

  • __prometheus_server: Installs and configures Prometheus server. Yay!

  • __prometheus_alertmanager: Installs and configures Prometheus Alertmanager. Yay too!

  • __grafana_dashboard: Installs and configures Grafana. Yay three!

PROMPORT=9090
ALERTPORT=9093

__daemontools
__golang_from_vendor --version 1.8.1  # required for prometheus and many exporters

require="__daemontools __golang_from_vendor" __prometheus_server \
--config "$__type/files/prometheus.yml" \
--retention-days 14 \
--storage-path /data/prometheus \
--listen-address "[::]:$PROMPORT" \
--rule-files "$__type/files/*.rules" \
--alertmanager-url"http://monitoring1.node.consul:$ALERTPORT,http://monitoring2.node.consul:$ALERTPORT"

require="__daemontools __golang_from_vendor" __prometheus_alertmanager \
--config "$__type/files/alertmanager.yml" \
--storage-path /data/prometheus \
--listen-address "[::]:$ALERTPORT"

__grafana_dashboard

Node exporter:

__daemontools
__golang_from_vendor --version 1.8.1  # required for prometheus and many exporters

require="__golang_from_vendor" __go_get github.com/prometheus/node_exporter
require="__daemontools" __daemontools_service node_exporter --run "/opt/gocode/bin/node_exporter"

Note: these types are currently available in cdist's master branch, but not yet in a release.

A cdist lightning-quick-start

git clone https://github.com/ungleich/cdist.git
cd cdist
$EDITOR cdist/conf/manifest/init
bin/cdist config -vv <hostname>

Read more at cdist quickstart.

Find Kamila’s article on github.