Operations & monitoring

Persistence, queues and limits, statistics, and logs.

Persistence

[storage]
backend = "sqlite"
path = "/var/lib/mercurio/mercurio.db"

Or, in a PostgreSQL database:

[storage]
backend = "postgres"
url = "postgres://mercurio:password@localhost/mercurio"

With backend = "memory", the default, everything is lost when the broker stops. With SQLite or PostgreSQL, the broker keeps:

• sessions and their subscriptions, with their expiry;
• retained messages, with the client that published them;
• wills, including those delayed with Will Delay Interval;
• messages queued for disconnected clients with persistent sessions, and messages in flight.

After a restart, clients with persistent sessions resume where they left off: their queued messages are delivered, and unacknowledged ones are sent again. Sessions that expired while the broker was down are discarded, and their wills published.

Changes are written in the background, in order for each client and in batches, so storage doesn't slow clients down. They reach the database within milliseconds: a broker that crashes loses only the changes of its last moments, and one that shuts down writes everything first. When storage falls behind, by 10,000 changes for one client, that client waits for it. Changes that fail because the database is unreachable, busy or out of connections are retried until it recovers; changes the database refuses are given up after five attempts, with an error in the log.

The SQLite database runs in WAL mode: while the broker runs, it keeps mercurio.db-wal and mercurio.db-shm next to the database, and recent changes may be only in the -wal file. To back it up while the broker runs, use sqlite3 mercurio.db ".backup copy.db"; a plain copy of mercurio.db alone can miss those changes. The broker creates the database readable by its owner only, and holds a lock on mercurio.db.lock while it runs: a second broker started on the same database exits with an error, while reading it with sqlite3, as for a backup, still works. The database records its schema version, and a broker refuses a database written by a newer version of it.

PostgreSQL needs version 12 or later. Connect the broker to it directly: PgBouncer drops the connection settings the broker relies on, and in transaction mode the lock described below. The broker creates its tables, and updates them when a new version needs it; it refuses a database updated by a newer version.

Standby brokers

Only one broker serves from a PostgreSQL database at a time, since two would overwrite each other's sessions. A broker started on a database another one is using exits with an error, unless it is a standby:

[storage]
backend = "postgres"
url = "postgres://mercurio:password@db.example.com/mercurio?sslmode=require"
standby = true

A standby waits without listening on its ports. It takes over as soon as the serving broker's process stops, or about 11 seconds after the database last heard from the serving broker's host, and serves every session from the database. Point clients at both brokers, through a load balancer or a list of addresses, so they reconnect to whichever serves.

A standby updates the tables only once it takes over, so one running a newer version can wait next to an older serving broker; a standby older than the tables exits at once. Its wait is not cut short by a statement_timeout, lock_timeout or idle_session_timeout set on the database role, as managed PostgreSQL services often do.

A broker that can't reach the database for about 7 seconds stops serving and exits with an error, because a standby may take over. With standby = true and the provided systemd unit, it restarts as the standby. Each takeover is recorded in the database, which refuses writes from brokers that served before it, even those delayed in the network, so a replaced broker can't overwrite the new one's sessions. A broker that has a write refused this way stops at once, as when it loses the database.

Queues and slow clients

Each session has its own queue, so a slow or offline client never makes others lose messages. When its queue is full, further messages for that client are dropped:

QoS 0 messages, which their publisher accepted could be lost, go first. The broker logs one warning when a queue starts dropping and another, with the count, once it has room again.

MQTT 3.x clients connecting without clean session keep their session forever. persistent_client_expiration bounds it, so devices that are reinstalled with a new client id don't leave sessions behind:

[server]
persistent_client_expiration = 604800  # one week

Protecting the broker

$SYS statistics

Every sys_interval seconds (10 by default), the broker publishes retained statistics under $SYS/broker/, with mosquitto's names, so dashboards built for mosquitto work unchanged. Retained $SYS messages, bridge states included, are kept in memory: a new subscriber gets the latest values, but they are not written to storage and don't count toward max_retained_messages or retained messages/count.

recado sub -t '$SYS/broker/#' -T

Clients may subscribe to $SYS topics, subject to ACLs, but never publish to them. Wildcards at the start of a filter, such as # or +/broker, don't match $SYS topics.

A stored session that can't be read when the broker starts, such as one written by a newer version before a downgrade, or a damaged row, doesn't keep the broker from starting. The broker logs an error naming the client and what failed, leaves the session's rows in storage as they are, and counts it in storage/sessions/set aside. Its client is refused with Server Unavailable until the session is deleted with mercurioctl client delete-session <client id>, after which it connects with a new session.

Health and metrics

A WebSocket listener with health = true answers GET /healthz with 200 OK while the broker serves, and 503 Service Unavailable to a request read as it starts stopping; once it has stopped, the listener refuses connections. With metrics = true it answers GET /metrics with the counters of $SYS in the Prometheus text format, read at the time of the request whatever sys_interval is. Both are off by default and need no http_dir.

Metrics tell anyone who can reach the listener how many clients the broker has and how much traffic it carries. They name no clients or topics, but enable them on a listener of their own on a loopback address, or one only your monitoring network reaches:

[[listeners]]
host = "127.0.0.1"
port = 9883
protocol = "websocket"
health = true
metrics = true

$ curl -s http://127.0.0.1:9883/metrics
# HELP mercurio_clients_connected Clients connected now
# TYPE mercurio_clients_connected gauge
mercurio_clients_connected 12
…
# HELP mercurio_messages_received_total PUBLISH packets received
# TYPE mercurio_messages_received_total counter
mercurio_messages_received_total 48210
…
# HELP mercurio_messages_dropped_total Messages lost before reaching a subscriber, by cause
# TYPE mercurio_messages_dropped_total counter
mercurio_messages_dropped_total{cause="queue_full_qos0"} 0
mercurio_messages_dropped_total{cause="queue_full"} 3
…
# HELP mercurio_uptime_seconds Seconds since the broker started
# TYPE mercurio_uptime_seconds gauge
mercurio_uptime_seconds 86400

The endpoints share the listener's limits for plain HTTP: an 8 KiB request head, read within connect_timeout, 32 responses at once and a minute per response.

Logging

[logging]
level = "info"
# file = "/var/log/mercurio/mercuriod.log"
# syslog = true
# connection_messages = false

• Under systemd, standard output goes to the journal: journalctl -u mercuriod -f.
• file appends to a file. Rotate it with logrotate's copytruncate, as the file stays open.
• syslog sends each line to the local syslog daemon through /dev/log, with facility daemon and a severity from the level.
• connection_messages = false drops the lines logged for every connection and disconnection, which dominate the log of a busy broker. They are logged under mercurio::connections, and a level naming it, such as "info,mercurio::connections=info", turns them back on, in the configuration or with mercurioctl broker log-level.
• level can be set per module: "info,mercurio_server::session=warn". An empty or invalid level means info, with a warning in the log. mercuriod -v logs at debug level.
• mercurioctl broker log-level debug changes the level of a running broker until it restarts; see Administration.

Shutdown

On SIGTERM or Ctrl+C, every client is sent a DISCONNECT saying the server is shutting down, and the broker waits up to five seconds for connections to close. With SQLite or PostgreSQL, persistent sessions and their queues are in the database, so a restart loses nothing a client was promised.

Memory

On Linux with glibc, the daemon limits the allocator to two arenas. Under heavy load with many threads, glibc otherwise keeps the memory each thread freed, and the broker's memory grows and stays high; with two arenas it stays flat, at no measurable cost in throughput. Memory use is otherwise dominated by queued messages, which the queue limits above bound.