Message Queues

AMQP

• AMQP Concepts - RabbitMQ

RabbitMQ

Concepts: RabbitMQ Connection vs Channel

• Channels: RabbitMQ Channels
• Queues: RabbitMQ Queues

RabbitMQ for Beginners - Exchanges, Routing Keys, Bindings

Configuration and authentication

Create users and virtual hosts:

RMQ_USERNAME=rmq_user
RMQ_PASSWORD=s3cr3tp4ssw0rd
RMQ_VHOST=rmq_vhost
rabbitmqctl add_user $RMQ_USERNAME $RMQ_PASSWORD
rabbitmqctl add_vhost $RMQ_VHOST
rabbitmqctl set_permissions -p $RMQ_VHOST $RMQ_USERNAME ".*" ".*" ".*"

The three permission patterns are configure, write, read, in that order, not the read/write order most people expect. Configure covers creating and deleting queues and exchanges, write covers publishing, read covers consuming. A consumer that only needs to drain an existing queue gets "" "" ".*".

These changes are written to RabbitMQ’s internal database under the node data directory (/var/lib/rabbitmq). In a container they only survive as long as that directory does: mount a volume there to keep users and vhosts across restarts and rebuilds.

The official docker image (RabbitMQ on Docker Hub) can import a definitions file on boot, but it is not automatic: dropping the file at /etc/rabbitmq/definitions.json is not enough, the config has to point to it. In rabbitmq.conf:

definitions.import_backend = local_filesystem
definitions.local.path = /etc/rabbitmq/definitions.json

Bake both the config and the definitions file into a derived image and the entities are created at startup. The path can also be a directory, in which case every file in it is imported.

Two things worth knowing about those keys. definitions.local.path is the newer spelling of the older load_definitions; both still work on 4.4, and this is old versus new syntax, not 3.x versus 4.x (the 3.13 and 4.x docs both document the definitions.* form). The new form needs both lines: on 4.4, definitions.local.path without definitions.import_backend stops the node at boot with definition import source is configured but definitions.import_backend is not set. The legacy load_definitions works on its own. The other backend is https, with definitions.https.url.

Import runs on every node at every boot, and it overwrites: a password or a permission changed at runtime is put back to the file’s value on the next restart. Set definitions.skip_if_unchanged = true to only import when the file contents actually change.

Definitions file (/etc/rabbitmq/definitions.json):

{
    "users":[
            {"name":"rmq_user","password_hash":"...","tags":""},
            {"name":"rmq_admin","password_hash":"...","tags":"administrator"}
    ],
    "vhosts":[
            {"name":"/"}
    ],
    "permissions":[
            ...
    ],
    "parameters":[],
    "policies":[
            ...
    ],
    "queues":[
            ...
    ],
    "exchanges":[
            ...
    ],
    "bindings":[
            ...
    ]
}

Generate a definitions file from a running broker

Rather than writing that JSON by hand, export it from a broker you already configured. Three ways, and the first one is the one to reach for:

# on the node itself, no management plugin needed
rabbitmqctl export_definitions /path/to/definitions.json

# over the HTTP API, needs the management plugin
rabbitmqadmin definitions export --file /path/to/definitions.json

# same thing by hand
curl -u user:password http://localhost:15672/api/definitions | jq

rabbitmqctl export_definitions is the only option that does not require the management plugin. The reverse operations are rabbitmqctl import_definitions and rabbitmqadmin definitions import.

The exported file contains password hashes: treat it as a secret, or strip the sensitive parts on the way out. rabbitmqadmin v2 can do that during the export:

rabbitmqadmin definitions export --stdout --transformations exclude_users,exclude_permissions

Note that rabbitmqadmin v2 is not the old Python script: it is a separate project shipped as a native binary (rabbitmqadmin-ng), with its own release cycle, and recent versions still talk to 3.12.x nodes. The management images ship v2 as rabbitmqadmin. The v1 Python script, whose command syntax differs, used to be downloaded from the node at /cli/rabbitmqadmin: 3.12 still serves it, 4.4 answers 404.

To reach the HTTP API of a broker running in Kubernetes, port-forward the management port first:

kubectl port-forward rabbitmq-54644f7685-gf8hw 15672:15672

Then open http://localhost:15672/ (default login guest:guest). Every official Docker image contains the management plugin, but only the management tags enable it (and bundle rabbitmqadmin).

Clients

• aio-pika Documentation
• aiormq - Pure Python AMQP client

Topology

Exchange to exchange bindings

Binding an exchange to another exchange, rather than to a queue, lets you fan out to several topologies without the publisher knowing about them.

• Exchange to Exchange Bindings - RabbitMQ
• Exchange to Exchange Bindings - RabbitMQ Blog
• Exchange to exchange bindings in RabbitMQ - CloudAMQP
• RabbitMQ Exchange to Exchange Bindings AMQP

Links & references

• Extract Transform Load, where a broker usually sits in a pipeline
• Kubernetes, for running a broker on a cluster