MQTT Performance Tests: Message Persistence

Persistence

Everything on the other test pages runs in memory. This page measures what it costs to make delivery survive a broker restart: every session is persistent, every message is QoS 1, and the record of each delivery is written to Redis before the message is handed on. Nothing is lost if the broker stops, and the question is what that guarantee is worth in throughput and latency.

The scenario is point-to-point, the same shape as the in-memory point-to-point page: each publisher has its own subscriber on its own topic, so the broker does no fan-out and what is measured is the per-message cost of routing plus the cost of persisting it. Both client groups connect with clean_session off, so a run at 20K holds 40,000 persistent sessions. Each publisher sends one message per second with a 16-byte payload. Redis runs on the broker host, and the broker starts against an empty database.

This configuration answers a particular need, and is not the one most deployments want. Persistence earns its cost where a message that never arrives is worse than a message that arrives late: commands to equipment that is intermittently connected, orders and transactions, anything a subscriber must receive even if it was offline when the message was sent, or if the broker restarted between the two. For live telemetry, metrics and status streams — where the next reading supersedes the last and a gap costs nothing — persistence buys nothing, and the in-memory figures on the other test pages are the ones that apply.

It is worth setting the numbers below against that. Workloads that genuinely need durable delivery are usually counted in messages per device per minute rather than per second, so the rates measured here are well clear of what they ask for. The ceiling matters when durable and high-volume traffic share one broker, and the two can be separated: only sessions connecting with clean_session off and messages sent at QoS 1 or above pay for persistence at all, so a broker can carry both without the fast traffic paying the durable traffic's cost.

Latency uses a logarithmic axis. A flat line is a broker keeping up; a line that climbs for the whole run is a backlog growing behind it, and is drawn dashed. Hover the chart for per-interval values.

Reading the results

• Fully durable delivery holds up to 30K messages/second at 330us average latency, with every message recorded in Redis before it is sent. Latency is flat across the run at 10K, 20K and 30K, which is what distinguishes a rate the broker sustains from one it merely survives.
• At 40K the broker is past its limit. The offered rate is still met — the load generator paces itself — but latency climbs from 847us at the first interval to 2.74ms at the last, without ever flattening. That is a queue filling faster than it drains: the 1.29ms average describes when the run stopped rather than any level the broker settles at, and a longer run would report a worse figure. It is listed for completeness, not as a result.
• Past the limit it degrades rather than collapses. The offered rate is still met, and the backlog costs single-digit milliseconds across a five-minute run — not the seconds, or tens of seconds, that an overloaded broker usually starts reporting. Being beyond the sustainable rate is not the same as falling over, and a short burst above it is absorbed rather than punished.
• Memory is not the constraint. Peak RSS grows with session count and stays small in absolute terms — 0.13 Gb at 10K rising to 0.45 Gb at 40K — so the limit is reached in CPU and round trips long before memory matters.
• Redis is close behind, but it is not what stops the broker first. Redis CPU plateaus at 85–94% of a single core from 20K onward rather than rising with the load, so its cost per message is falling as the rate climbs. It is single-threaded, so that plateau is near its ceiling and it becomes the next limit as soon as the one below is lifted.

Where the cost goes, and how to trade it

By default XMQ is fully durable: a QoS 1 message for a persistent session waits for its Redis record before it is delivered. Because the sending thread is parked for that round trip, the writes never overlap — one message, one round trip — and that, rather than the amount of work Redis does, is what sets the ceiling above.

The max_queued_writes setting is the dial. Left at 0 it is the fully durable behaviour measured here. Set above 0, deliveries proceed while their records are written, so the writes pipeline and the ceiling moves up considerably; the number is the count of messages that may be in flight without a durable record, and so exactly how many could be lost if the broker were killed at the wrong instant. It is a deliberate, bounded trade rather than a hidden one, which is why the durable configuration is what gets published here.

For comparison, the same hardware and the same point-to-point shape without persistence sustains over 100,000 messages/second at 278us (see Point-to-point). Durability is therefore worth roughly a threefold reduction in throughput — and that is a lower bound, since the in-memory run had not reached its own limit.