Monitoring

dockerbashcontainer

Generic / multipurpose

Prometheus, Alertmanager and Grafana - the open-source Prometheus + Alertmanager + Grafana stack (pull-based, self-hosted).
Datadog: https://www.datadoghq.com/ (SaaS, ~15 $ / month / host)

Status page applications

Statuspage: https://www.statuspage.io/ (SaaS, 30 to 400 $ / month)
Cachet: https://github.com/cachethq/Cachet (PHP)
Stashboard: https://github.com/twilio/stashboard (Python on GAE)

Software

New relic: http://newrelic.com/ (SaaS, 50 to 150 $ / month, free lite version available too)
Sentry: https://github.com/getsentry/sentry (Python/Django)
Hosted Sentry: https://getsentry.com/welcome/ (SaaS, 20 to 200 $ / month)

Datadog

Pretty standard monitoring app, run as a service. Possible to run the agent in a privileged container.

CoreOS unit to run agent

[Unit]
Description=Monitoring Service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill dd-agent
ExecStartPre=-/usr/bin/docker rm dd-agent
ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
ExecStart=/usr/bin/bash -c \
  "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /proc/mounts:/host/proc/mounts:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -v /home/core/dd-agent:/etc/dd-agent:ro \
  -e API_KEY=XXX_API_KEY_HERE_XXX \
  datadog/docker-dd-agent"

[X-Fleet]
Global=true

Add etcd watcher

Grab default config to host (needed to mount the volume above anyway):

docker cp dd-agent:/etc/dd-agent .

dd-agent/conf.d/etcd.yaml:

init_config:

instances:
  - url: "http://10.1.65.1:2379"
    timeout: 5

To find the host IP address useable in the agent container, exec a bash and run ip route, the default gateway is the host.

To check if we’re good:

docker exec dd-agent service datadog-agent info