CoreOS

nginxdockerhttpssh

Note: kept for the archives. CoreOS Container Linux reached end of life on May 26, 2020 (Fedora CoreOS and Flatcar picked up the torch), and fleet was deprecated back in 2017 in favor of Kubernetes. Nothing below is actionable today.

Issues

Admin console (locally, or out of the cluster):

$ fleetctl start instances/*
2015/06/10 22:05:44 WARN fleetctl.go:772: Error retrieving Unit(nginx-discovery@3.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx-discovery%403.service?alt=json: forwarding request denied
2015/06/10 22:05:44 WARN fleetctl.go:772: Error retrieving Unit(nginx@1.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%401.service?alt=json: forwarding request denied
2015/06/10 22:05:44 WARN fleetctl.go:772: Error retrieving Unit(nginx@2.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%402.service?alt=json: forwarding request denied
2015/06/10 22:05:44 WARN fleetctl.go:772: Error retrieving Unit(nginx@3.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%403.service?alt=json: forwarding request denied
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx-discovery@2.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx-discovery%402.service?alt=json: ssh: rejected: administratively prohibited (open failed)
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx@1.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%401.service?alt=json: forwarding request denied
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx@2.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%402.service?alt=json: forwarding request denied
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx@3.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%403.service?alt=json: forwarding request denied
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx-discovery@2.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx-discovery%402.service?alt=json: ssh: rejected: administratively prohibited (open failed)
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx@2.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%402.service?alt=json: ssh: rejected: administratively prohibited (open failed)
2015/06/10 22:05:45 WARN fleetctl.go:772: Error retrieving Unit(nginx@3.service) from Registry: Get http://domain-sock/fleet/v1/units/nginx%403.service?alt=json: ssh: rejected: administratively prohibited (open failed)

In journal:

Jun 10 19:58:21 core3.c.rdorgueil.internal sshd[1341]: no more sessions
Jun 10 19:58:25 core3.c.rdorgueil.internal fleetd[1386]: ERROR reconcile.go:79: Unable to determine agent's current state: failed fetching unit states from UnitManager: Unit name nginx-discovery@.service is not valid.
Jun 10 19:58:35 core3.c.rdorgueil.internal fleetd[1386]: ERROR reconcile.go:79: Unable to determine agent's current state: failed fetching unit states from UnitManager: Unit name nginx@.service is not valid.

Only solution found (https://github.com/coreos/fleet/issues/969) is to reboot faulty instances after destroying the faulty units.

docker run -it --link postgres.1:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'

A global unit: the Datadog agent on every node

Global=true in the X-Fleet section tells fleet to schedule the unit on every machine of the cluster, which is what a monitoring agent wants. The ExecStartPre lines prefixed with - may fail without failing the unit, so the same file works on first start and on restart.

[Unit]
Description=Monitoring Service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill dd-agent
ExecStartPre=-/usr/bin/docker rm dd-agent
ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent
ExecStart=/usr/bin/bash -c \
  "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /proc/mounts:/host/proc/mounts:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -v /home/core/dd-agent:/etc/dd-agent:ro \
  -e API_KEY=XXX_API_KEY_HERE_XXX \
  datadog/docker-dd-agent"

[X-Fleet]
Global=true

The agent configuration is mounted from the host, so the directory has to exist first. Copying it out of the image is the quickest way to get a valid one:

docker cp dd-agent:/etc/dd-agent .

Adding a check is then a file in conf.d, here dd-agent/conf.d/etcd.yaml watching the local etcd:

init_config:

instances:
  - url: "http://10.1.65.1:2379"
    timeout: 5

The address is the host as seen from inside the container: run ip route in the container, the default gateway is the host. To check the agent picked the check up:

docker exec dd-agent service datadog-agent info

That image (Agent 5), the API_KEY variable and the info command are all gone from current Datadog agents. The command that works today is in Monitoring.

Local cluster tools

The above tools have a lot of problems on my laptop, but the idea sounds promising.

Related