High-Availability Clustering - Coreflux Documentation

High-availability clustering is in beta. It has been tested by repeatedly killing, restarting, and isolating nodes under load, but not yet in production deployments, at production message rates, or across rolling upgrades. Its behavior and configuration options can change in a later release. Test it with your own workload before you rely on it in production, and include your broker version when you report an issue.

Keep messaging available when a node fails

A single broker is a single point of failure. High-availability clustering (beta) runs several Coreflux MQTT Broker nodes as one system. Clients connect to any node, and a message published on one node reaches subscribers on every node. If a node stops, the others keep serving clients and take over its work. A QoS 1 or 2 message is acknowledged to the publisher only once it is stored on a majority of the nodes responsible for it, so losing a minority of the cluster does not lose messages the broker has already accepted.

Like a team that keeps shared minutes. One member chairs the meeting, and every decision is written into the minutes of most members before it counts. If the chair leaves, the others pick a new one and carry on from the same minutes.

When to use clustering

Use clustering when you need:

MQTT service that continues when one broker stops, crashes, or is restarted for maintenance;
QoS 1/2 messages that survive the loss of a node after the broker has acknowledged them; or
one MQTT endpoint served by several broker nodes, on fixed machines or as Kubernetes pods.

Run 3 or 5 nodes. A cluster keeps working as long as a majority of its nodes are up: 2 of 3, or 3 of 5. Clustering requires a license that grants the cluster feature; with clustering enabled and no such license, the broker logs a warning and runs standalone.

Clustering is broker configuration, not a LoT Connector. To exchange topics with another, separate MQTT broker (including non-Coreflux brokers), use an MQTT Bridge instead.

Setting up a cluster takes the same steps on every node: a service user for traffic between nodes, a Cluster section in the broker configuration, and durability turned on. Then load your LoT on any node, and the cluster replicates it to the others. See LoT in a cluster.

Quick Start

Create the cluster service user on every node

Nodes exchange messages with each other over MQTT, using a dedicated MQTT user. Create the same user, with the same password, on every node, for example with the -addUser broker command:

-addUser cluster a-strong-password

Only this user may send cluster traffic between nodes.

Configure the cluster on every node

Add a Cluster section to the broker configuration of each node, or set the matching COREFLUX_CLUSTER_* environment variables (environment variables override the file). Keep everything identical except NodeName, and turn durability on as well (see Durability and clustering).

Seeds (typical production)
Kubernetes
Multicast (same LAN)
Environment variables (Docker)

List the peers’ addresses when automatic discovery is not available, which is the case in most cloud networks and across subnets:

{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "factory-floor",
    "NodeName": "broker-a",
    "Discovery": "seeds",
    "Seeds": ["broker-a:9200", "broker-b:9200", "broker-c:9200"],
    "SharedSecret": "a-long-random-secret",
    "DataPlaneUsername": "cluster"
  },
  "Durability": {
    "Enabled": true
  }
}

Discover the pods through a headless Service. Use the StatefulSet pod name as NodeName, so each node keeps its identity when its pod is rescheduled:

{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "k8s-brokers",
    "NodeName": "coreflux-broker-0",
    "Discovery": "kubernetes",
    "KubernetesServiceDns": "coreflux-broker-headless.default.svc.cluster.local",
    "KubernetesPeerPort": 9200,
    "SharedSecret": "a-long-random-secret",
    "DataPlaneUsername": "cluster"
  },
  "Durability": {
    "Enabled": true
  }
}

Let nodes on one local network find each other automatically:

{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "lab-cluster",
    "NodeName": "broker-1",
    "Discovery": "multicast",
    "SharedSecret": "a-long-random-secret",
    "DataPlaneUsername": "cluster"
  },
  "Durability": {
    "Enabled": true
  }
}

The same seeds setup for a container, without editing the configuration file:

environment:
  COREFLUX_CLUSTER_ENABLED: "true"
  COREFLUX_CLUSTER_ID: "factory-floor"
  COREFLUX_CLUSTER_NODE_NAME: "broker-a"
  COREFLUX_CLUSTER_DISCOVERY: "seeds"
  COREFLUX_CLUSTER_SEEDS: "broker-a:9200,broker-b:9200,broker-c:9200"
  COREFLUX_CLUSTER_SHARED_SECRET: "a-long-random-secret"
  COREFLUX_CLUSTER_DATAPLANE_USERNAME: "cluster"
  COREFLUX_DURABILITY_ENABLED: "true"

Start the nodes

Start all nodes. On a first start, the nodes wait up to BootstrapGraceSeconds (30 s by default) to make sure no cluster is running yet, then form a new one and elect a leader. Later restarts rejoin the existing cluster at once.

Verify the cluster

Call GET /cluster on the operations port of any node (9100 by default):

curl http://broker-a:9100/cluster

Every node names the same leadership.leader, membersTotal equals your number of nodes on every node, and on the leader membersAlive equals membersTotal.

How a cluster works

One leader, automatic membership

The nodes elect one leader. The leader admits new nodes, removes dead ones, and decides which node is responsible for which part of the message traffic. Every node, leader or not, serves MQTT clients in the same way.

Elections are automatic. When the leader is lost, the remaining nodes elect a new one within a few election timeouts, typically under a second. That includes a leader that is silent: frozen, or cut off so its cluster port no longer answers.
Nodes keep their identity. A node is recognised by its NodeName, not by its IP address. A node that restarts on a new address, or on another node’s old address, is recognised and its old address is dropped at once.
Restarts rejoin. Each node remembers its cluster membership on disk and rejoins its peers after a restart, including when every node restarts at the same time.
Dead nodes are removed. A node that stays unreachable for MemberRemovalGraceSeconds (5 minutes by default) is removed from the membership, so it no longer counts when the cluster checks for a majority. With kubernetes discovery, a pod that the headless Service no longer lists, and whose cluster port fails a TCP probe, is removed after MemberAbsentFromDiscoverySeconds (2 minutes, 120 seconds, by default). A new pod is found within seconds. A node is added back when it returns.
LoT changes go through the leader. Send them to any node: a follower passes them to the leader. Other commands, such as user management, run on the node that receives them. See LoT in a cluster.
No accidental second cluster. A node that starts with no cluster state first asks its peers whether they already belong to a cluster, and joins that cluster instead of starting a new one.

Partitions spread the work

QoS 1/2 traffic is split into partitions (32 by default). A publisher’s messages always belong to the same partition, chosen from its client id. Each partition has one owner, the node that accepts its writes, and copies on other nodes (3 copies by default, owner included). The leader spreads the owners evenly across the nodes. When a node fails, its partitions move to nodes that already hold their copies, so no accepted message is left behind.

QoS 1 and 2 are acknowledged once they are safe

When a client publishes at QoS 1 or 2, the broker stores the message on a majority of its partition’s copies (2 of 3) before it sends the PUBACK (QoS 1) or PUBREC (QoS 2). A client connected to a node that does not own its partition is served the same way: that node passes the message to the owner and acknowledges it when the owner confirms. If a majority cannot be reached in time (ReplicationTimeoutMs, 5 s by default), the broker rejects the publish instead of acknowledging a message it cannot keep safe, and does not deliver it:

Reason code (MQTT 5)	Meaning	What the client should do
`0x97` Quota exceeded	Not enough nodes available for this partition right now (failover in progress, this side of a network split has no majority, or replication is still starting on the receiving node)	Retry after a short delay, on the same or another node
`0x80` Unspecified error	The node could not write the message to its disk	Retry; check the node’s disk

While replication is still starting, that 0x97 refusal uses the reason text “Cluster starting” and lasts at most 30 seconds. Retry; the node accepts the publish once replication is running.

Use MQTT 5 for QoS 1/2 publishers. MQTT 3.1.1 has no way to report a rejected publish: a 3.1.1 client receives a normal acknowledgement for a message the cluster did not accept. MQTT 5 clients see the reason code and can retry.

Subscribers on any node receive every message

A message published on one node is delivered to matching subscribers on every node:

QoS 1/2 messages are queued on disk for each other node and re-sent until that node confirms delivery. A node that is down receives its messages when it returns, including messages for its offline persistent sessions.
QoS 0 messages are sent to the other nodes once, without storage. A node that is down misses them, as with any QoS 0 message.

Failover

When a node fails, its partitions move to surviving nodes that already hold their copies. The new owner first catches up from the other copies, then re-delivers the messages of the last ReplicationReplaySeconds (5 minutes by default) for those partitions, so subscribers do not miss messages the failed node had accepted but not yet passed on. Subscribers can therefore receive some messages twice after a failover. With ReplicationDedupDelivered on (the default), a node drops a replay copy of a quorum-durable message it has already delivered. Set it to false to deliver every copy. During a failover, publishes to the affected partitions are rejected with 0x97 until the new owner is ready. A new leader is elected within a few election timeouts, typically under a second, including when the lost node is silent. Expect about 15 seconds after a node failure before that owner is ready, and up to about 25 seconds when the failed node was the leader. Publishes to partitions owned by the surviving nodes continue meanwhile. Clients that retry on 0x97 lose nothing.

Sessions follow the client

With durability enabled, a client with a persistent session that reconnects to another node gets its subscriptions back. Messages already queued for it while it was offline stay on the node it was connected to, and are delivered only if it reconnects to that node. See Limitations.

Durability and clustering

Clustering and MQTT state durability protect different parts of a message’s path, and a production cluster needs both:

Clustering protects the message after the publisher’s acknowledgement: it is stored on a majority of nodes and survives the loss of one of them.
Durability protects the subscriber side on each node: persistent sessions, their subscriptions, the messages queued for them, and retained messages survive a restart of that node.

What is lost when …	Clustering only	Clustering + durability
a node with persistent subscribers restarts	the messages queued for them, and the messages published while the node was down	nothing
a node that owns partitions fails	nothing	nothing
a client reconnects to another node	its subscriptions	the messages queued for it on the old node (delivered only if it reconnects there)
a node restarts (retained messages)	the node’s retained messages	nothing

Enable durability on every node of a cluster. Clients without persistent sessions (MQTT 3.1.1 clean session, MQTT 5 session expiry 0) are not affected either way.

LoT in a cluster

The cluster runs LoT as one logical broker. A client can connect to any node and sees the same definitions, the same shared state, and the same results. A broker that isn’t clustered behaves as before.

The three rules

Same definitions everywhere. Every node runs the same version of every action, model, connector, rule, Python script, and panel. Changes go through the leader and are replicated to every node.
One place per piece of work. A message runs LoT on the node it entered. A client event runs on the node that holds the connection. Timers, startup, and device connections run on one owner node that the cluster picks.
A message from a peer never triggers LoT. It still reaches that node’s subscribers and caches, but the action that reacts to it runs once, where the message came in.

Where LoT runs

Trigger	Examples	Runs on
A message	`ON TOPIC`, `ON CHANGE`, a model’s trigger fields	The node the message entered. The output reaches subscribers on every node
A client event	`ON CONNECT`, `ON DISCONNECT`, `ON SUBSCRIBE`	The node holding the connection
A timer	`ON EVERY`, `ON EVERY WEEKDAY AT`	The action’s owner
Startup	`ON START`	The action’s owner, once per cluster session. See ON START and KEEP
Called code	`CALL ACTION`, `CALL PYTHON`	Wherever the caller runs. Python module state stays on that node
Access rules	`DEFINE RULE`	Every node
Shared state	`KEEP TOPIC`	Where the action runs. The value reaches every node

Where connectors run

Class	Connector types	Runs on
Client of a service	PostgreSQL, MySQL, MariaDB, SQL Server, MongoDB, OpenSearch, CrateDB, Snowflake, REST client, email, LLM, MCP, barcode	Every node. Events run where the message entered; a scheduled event (`WITH EVERY`) has one owner
Exclusive connection	Modbus TCP and serial, Siemens S7, OPC UA client, EtherNet/IP, ADS, Allen-Bradley, FINS, BACnet, Sparkplug host and edge node, MQTT bridge to a remote broker, video input, audio input, `FILE_STORAGE`, `MQTT_TO_REST`, streaming gRPC	Only its owner. Other nodes hold the definition and report it as standby
Server	`MODBUS_SERVER`, `OPCUA_SERVER`, `VIDEO_OUTPUT`, `REST_API` in server mode	Every node, on `0.0.0.0:<port>`. See Server connectors
Group consumer	`KAFKA`	Every node, in one consumer group. An `"in"` Kafka connector must set `GROUP_ID`, or every node reads every message
Local transform	MQTT bridge from `SELF` to `SELF`, Sparkplug decode	Every node

A MODBUS_SERVER on a serial port is an exclusive connection: only its owner opens the port. A TRIGGER or STORE IN on a node that doesn’t own the connector is sent to the owner and runs there. A FETCH needs the answer where it was asked, so on another node it fails with a message that names the owner. Use an event that writes to a DESTINATION_TOPIC instead.

Pin a connector

To keep an exclusive connector on the node that is cabled to the device, set CLUSTER_NODE in CONNECTOR_CONFIG (ROUTE_CONFIG remains a supported alias) to that node’s NodeName:

DEFINE CONNECTOR Press1 WITH TYPE MODBUS_TCP
    ADD MODBUS_CONFIG
        WITH CONNECTION_TYPE "TCP"
        WITH IP "192.168.1.100"
        WITH PORT 502
        WITH SLAVE_ID 1
    ADD CONNECTOR_CONFIG
        WITH CLUSTER_NODE "coreflux-node2"
    ADD MAPPING Poll
        WITH EVERY 1 SECONDS
        WITH SOURCE_TOPIC "press1/read"
        ADD TAG Temperature
            WITH ADDRESS "0"
            WITH ADDRESS_TYPE "HOLDING_REGISTER"
            WITH DATA_TYPE "INT16"

While the pinned node is down, the connector runs nowhere. Leave CLUSTER_NODE out to let the cluster place and balance it.

Change definitions

Send LoT commands (-addAction, -removeConnector, -loadProject, MCP tools, the HUB) to any node. A follower passes the command to the leader, which validates and commits it. The follower answers once its own node has applied the change.

A command that fails changes nothing on any node. A project load is applied as one change.
Changes are refused, with the reason, while a majority of nodes can’t be reached, or while a live node runs a version without replicated LoT. Upgrade every node before you change LoT. lot.mixedMode in GET /cluster shows when this is the case.
Users and credentials are not replicated. Create them on every node: the node that receives a command checks your permission locally.

First start. Until the first change is committed, each node runs its own LoT. Once the leader has been stable for a few seconds, or on the first LoT command, the leader’s definitions become the cluster’s. A node whose files differ drops what the cluster doesn’t have and saves each definition it removes or replaces under lot-backup/<timestamp>/<kind>/<name>.lot. A node that was away catches up when it returns: it applies what it missed, removes what the cluster removed, and backs up the files it drops.

Owners and failover

Each timer, ON START action, exclusive connector, and scheduled event has one owner and one standby node. Every node agrees on the map, and the cluster spreads the work over the live nodes.

Event	What happens
A node stops or is cut off	After 6 s of silence, its work moves to the standby nodes. Nothing else moves
A node comes back	It takes new work, not work that is already running elsewhere
An owner loses contact with a majority	Within 4 s it stops its timers and closes its device connections itself, before another node takes them over, so a device never sees two clients
The leader fails	The new leader continues from the committed map

Plan on about 10 seconds without a timer or device connection when its owner fails. After a takeover:

A timer starts a new cycle, and runs missed during the takeover aren’t made up.
DATE_TIME timers use the node’s local time, so set the same time zone on every node.
Python module globals, LLM conversations, and the OT publish-on-change state start fresh on the new owner.
The node that loses a connector doesn’t clear its tag topics.

ON START and KEEP

ON START runs once per cluster session, on the action’s owner, and again when the action is redeployed with different code.

A session starts when the cluster forms from nothing and lasts while any of its members is alive. Restarting one node, or all but one, runs nothing. Restarting every node starts a new session.
An ON START action runs at most once per session and code version, even if its node fails before it runs.
While a definition fails to apply on a node, ON START actions wait on that node.

KEEP TOPIC values are sent to every node’s cache, without reaching subscribers or triggering LoT there.

Values are eventually consistent: when two nodes keep the same topic at about the same time, the later write wins. A normal publish to the topic replaces a kept value.
A node that was down gets the values when it returns, and a node keeps its values across a restart. Each value can be up to 1 MB, with up to 100,000 kept topics per node.

Server connectors

A server connector (Modbus server, OPC UA server, REST API server, video output) runs on every node, so a load balancer or Kubernetes Service can send a client to any of them. Every node serves the current value of every tag.

OPC UA behind one name. Set WITH ADVERTISED_HOST "opcua.example.com" in OPCUA_SERVER_CONFIG (a host name or address, no scheme, port, or path). To present one identity, put the same PKCS#12 certificate with its private key in <PKI_DIR>/own/private on each node, or import it with -connectorCertificates import. In a container, set PKI_DIR to a writable folder.
Failed listeners are reported. GET /cluster lists every connector listener under routeListeners with its state, also on retained $SYS/Coreflux/Cluster/RouteListeners and as coreflux_route_listener_up on /metrics. A failed bind isn’t retried.
Open every port. The Service or load balancer must list every server connector port, including ones you add later. MODBUS_SERVER defaults to port 502, which a non-root container can’t bind: map it to an unprivileged port.

What is guaranteed

A LoT run is at most once per message. If a node fails after a message is accepted but before LoT runs, that run is lost. Failover replay delivers the message again but doesn’t run LoT again.
LoT output isn’t quorum-durable. It’s forwarded to the other nodes over the reliable link.
ON CHANGE compares against the last value this node saw. There is no cluster-wide order per topic.
QoS 1 redelivery runs an ON TOPIC action twice, as on a single broker.
Definitions match on every node once a change is applied. Until then, lot in GET /cluster shows which node is behind and why.

Configuration reference

Set these options in the Cluster section of the broker configuration, or with the listed environment variables (environment variables win). Use the same values on every node, except NodeName, AdvertisedHost, and BindIP.

Identity and discovery

Config field	Env var	Default	Description
`Enabled`	`COREFLUX_CLUSTER_ENABLED`	`false`	Turns clustering on. The license must also grant the cluster feature
`ClusterId`	`COREFLUX_CLUSTER_ID`	`coreflux-cluster`	Name of the cluster. Required
`NodeName`	`COREFLUX_CLUSTER_NODE_NAME`	host name	This node’s identity in the cluster. Must be unique and must not change; in Kubernetes, use the StatefulSet pod name
`Discovery`	`COREFLUX_CLUSTER_DISCOVERY`	`multicast`	How nodes find each other: `seeds`, `kubernetes`, or `multicast`
`Seeds`	`COREFLUX_CLUSTER_SEEDS`	—	Peer addresses as `host:port`, separated by `,`, `;`, or spaces. Required with `seeds`
`KubernetesServiceDns`	`COREFLUX_CLUSTER_K8S_SERVICE_DNS`	—	DNS name of the headless Service. Required with `kubernetes`
`KubernetesPeerPort`	`COREFLUX_CLUSTER_K8S_PEER_PORT`	`Port`	Cluster port of the pods found through that Service
`MulticastGroup`	`COREFLUX_CLUSTER_MULTICAST_GROUP`	`239.63.114.71`	Multicast group for `multicast` discovery
`MulticastPort`	`COREFLUX_CLUSTER_MULTICAST_PORT`	`9201`	UDP port for `multicast` discovery
`SharedSecret`	`COREFLUX_CLUSTER_SHARED_SECRET`	—	Signs all traffic between nodes so it cannot be forged; with `multicast`, also keeps foreign nodes out. Set it in production; the broker logs a warning when it is missing

Network

Config field	Env var	Default	Description
`BindIP`	`COREFLUX_CLUSTER_BIND_IP`	`0.0.0.0`	Local address of the cluster port
`Port`	`COREFLUX_CLUSTER_PORT`	`9200`	Cluster TCP port (leader election and membership)
`AdvertisedHost`	`COREFLUX_CLUSTER_ADVERTISED_HOST`	`NodeName`	Address the other nodes use to reach this node. Set it when the node’s own address is not reachable from its peers (containers, NAT)
`AdvertisedPort`	`COREFLUX_CLUSTER_ADVERTISED_PORT`	`Port`	Port the other nodes use to reach this node
`DataPlaneUsername`	`COREFLUX_CLUSTER_DATAPLANE_USERNAME`	—	The cluster service user. It must exist, with a password, on every node. Without it, nodes do not exchange messages or LoT changes. If it is set but missing from a node’s users, clustering does not start on that node
`DataPlaneMqttPort`	`COREFLUX_CLUSTER_DATAPLANE_MQTT_PORT`	broker MQTT port	MQTT port nodes use to reach each other
`DataPlaneEnabled`	`COREFLUX_CLUSTER_DATAPLANE_ENABLED`	`true`	Forwards QoS 0 messages between nodes. When `false`, QoS 0 messages stay on the node they were published on
`Topology`	`COREFLUX_CLUSTER_TOPOLOGY`	`mesh`	How QoS 0 messages travel: `mesh` (every node to every node, recommended) or `ring` (to two neighbours only; complete only for up to 3 nodes)
`UseTls`	`COREFLUX_CLUSTER_TLS`	`false`	Uses TLS for the QoS 0 links between nodes. It does not encrypt the cluster port or the QoS 1/2 links (see Ports and network)
`DataDir`	`COREFLUX_CLUSTER_DATA_DIR`	`cluster` folder in the broker data path	Where the node keeps its cluster state and partition copies. Must be on persistent storage
`PublishInternalRoutes`	`COREFLUX_CLUSTER_PUBLISH_INTERNAL_ROUTES`	`false`	Troubleshooting aid: publishes the connectors the cluster generates between nodes, password redacted, on `$SYS/Coreflux/Cluster/DataPlane/Routes/{name}`

Replication

Config field	Env var	Default	Description
`ReplicationEnabled`	`COREFLUX_CLUSTER_REPLICATION_ENABLED`	`true`	Stores QoS 1/2 messages on several nodes, delivers them reliably to other nodes, and lets sessions follow clients. When `false`, QoS 1/2 messages travel between nodes like QoS 0, and a follower can’t pass LoT changes or connector calls to other nodes
`QuorumGateEnabled`	`COREFLUX_CLUSTER_QUORUM_GATE`	`true`	Waits for a majority before acknowledging QoS 1/2 publishes. When `false`, publishes are acknowledged once the receiving node has them, and are no longer protected against that node’s loss
`ReplicationFactor`	`COREFLUX_CLUSTER_REPLICATION_FACTOR`	`0` (= 3)	Copies of each partition, owner included (0–7), limited by the number of nodes up. `1` gives no protection across nodes
`PartitionCount`	`COREFLUX_CLUSTER_PARTITION_COUNT`	`32`	Number of partitions (1–4096). Choose it once: changing it on a running cluster reassigns every client
`ReplicationTimeoutMs`	`COREFLUX_CLUSTER_REPLICATION_TIMEOUT_MS`	`5000`	How long a publish may wait for a majority before it is rejected (100–60000)
`ReplicationWindow`	`COREFLUX_CLUSTER_REPLICATION_WINDOW`	`1024`	QoS 1/2 publishes that may wait for a majority at the same time on a node (1–65536). Beyond it, publishes are rejected at once with `0x97`
`ReplicationReplaySeconds`	`COREFLUX_CLUSTER_REPLICATION_REPLAY_SECONDS`	`300`	Messages re-delivered after a failover, in seconds (0 = none, up to 86400). Longer means more duplicates, shorter means more risk of a gap
`ReplicationForwardQueueEntries`	`COREFLUX_CLUSTER_REPLICATION_FORWARD_QUEUE_ENTRIES`	`8192`	QoS 1/2 messages kept for a node that is down (64–1000000). Beyond it, the oldest are dropped for that node
`ReplicationCompactEveryRecords`	`COREFLUX_CLUSTER_REPLICATION_COMPACT_EVERY`	`50000`	Compacts a partition’s stored messages after this many records, to bound disk use (1000–10000000)
`ReplicationDedupDelivered`	`COREFLUX_CLUSTER_REPLICATION_DEDUP_DELIVERED`	`true`	The node remembers which quorum-durable messages it delivered and drops failover-replay copies of them. `false` delivers every copy
`ReplicationTrace`	`COREFLUX_CLUSTER_REPLICATION_TRACE`	`false`	Records each QoS 1 or 2 message on the node under `replication/trace`. Read it with `GET /cluster/trace` on the ops port (`9100`). About 50 MB per hour per node at 100 messages per second
`ReplicationTraceMaxMegabytes`	`COREFLUX_CLUSTER_REPLICATION_TRACE_MAX_MB`	`512`	Cap on those trace files, in megabytes (8–8192). The oldest files are deleted first

Membership and startup

Config field	Env var	Default	Description
`BootstrapGraceSeconds`	`COREFLUX_CLUSTER_BOOTSTRAP_GRACE_SECONDS`	`30`	How long a node with no cluster state waits to be added to a running cluster before it may start a new one (1–3600). Also how long a node retries resolving its `AdvertisedHost` and looking for peers that moved
`BootstrapPeerCheck`	`COREFLUX_CLUSTER_BOOTSTRAP_PEER_CHECK`	`true`	Before starting a new cluster, asks each peer’s operations endpoint whether it already belongs to one. A peer that doesn’t answer counts as one that might, so the node keeps waiting. Turn off only if the operations ports cannot be reached between nodes
`PeerOpsPort`	`COREFLUX_CLUSTER_PEER_OPS_PORT`	`0` (= this node’s)	Operations port of the peers, for that check
`MemberRemovalGraceSeconds`	`COREFLUX_CLUSTER_MEMBER_REMOVAL_GRACE_SECONDS`	`300`	The leader removes a node that stays unreachable for this long from the membership, so it no longer counts toward the majority (0–86400). It is added back once discovery lists it and it answers. A member that Kubernetes discovery has stopped listing is removed with `MemberAbsentFromDiscoverySeconds` instead. `0` turns off removal of an unreachable node
`MemberAbsentFromDiscoverySeconds`	`COREFLUX_CLUSTER_MEMBER_ABSENT_FROM_DISCOVERY_SECONDS`	`120`	With Kubernetes discovery, the leader removes a member that DNS has stopped listing and that fails a TCP probe, after this many seconds (10–86400)
`MemberAddResignSeconds`	`COREFLUX_CLUSTER_MEMBER_ADD_RESIGN_SECONDS`	`60`	The leader resigns if it has tried this long to add a member that discovery lists and that answers a TCP probe, without the member joining (0 = never; 15–3600). The next leader tries again
`ElectionTimeoutLowerMs`	`COREFLUX_CLUSTER_ELECTION_LOWER_MS`	`150`	Lower bound of the randomised election timeout. Lower values detect a lost leader sooner
`ElectionTimeoutUpperMs`	`COREFLUX_CLUSTER_ELECTION_UPPER_MS`	`300`	Upper bound; must be greater than the lower bound
`ElectionRpcTimeoutMs`	`COREFLUX_CLUSTER_ELECTION_RPC_TIMEOUT_MS`	`0` (= 2/3 of the lower election timeout, 100 ms)	How long a node waits for a vote, pre-vote, or heartbeat reply, and to connect. Must be shorter than `ElectionTimeoutLowerMs`; otherwise clustering does not start
`HeartbeatThreshold`	`COREFLUX_CLUSTER_HEARTBEAT_THRESHOLD`	`0.25`	The leader sends a heartbeat every election timeout multiplied by this value (0.05–0.5)
`RaftTransport`	`COREFLUX_CLUSTER_RAFT_TRANSPORT`	`Deadline`	`Deadline` gives votes, pre-votes, and heartbeats their own reply deadline. `Tcp` is the fallback

The broker checks these values at start. If one is invalid, clustering does not start, the log names the problem, and GET /cluster reports enabled: false.

Ports and network

Port	Protocol	Purpose	Open between
`9200` (`Port`)	TCP	Leader election and membership	All nodes
Broker MQTT port (`1883` by default)	TCP	Messages between nodes, as the cluster service user	All nodes
`9100` (operations port)	TCP	Bootstrap check and monitoring	All nodes, and your monitoring system
`9201` (`MulticastPort`)	UDP	Discovery, `multicast` mode only	All nodes

Keep the cluster port on a private network. Traffic on port 9200 is neither encrypted nor authenticated. SharedSecret signs the messages exchanged between nodes, but not this port. Allow it only between cluster nodes.

Delivery guarantees

QoS	With clustering
0	Delivered to subscribers on every node that is up. Lost for a node that is down, or if the publishing node fails, as with any QoS 0 message
1	Acknowledged only once stored on a majority of its partition’s copies; otherwise rejected (`0x97`/`0x80`) and not delivered. Delivered to subscribers on every node, including nodes that are down when it is published (when they return). At least once: after a failover, some messages can arrive twice
2	The same as QoS 1. Between a client and its node the QoS 2 exchange works as usual, but across a node failure delivery is at least once, not exactly once

Design consumers to tolerate duplicates, for example with a message id or sequence number in the payload. For new designs, QoS 1 gives the same guarantee as QoS 2 in a cluster, with one round trip less.

Operations

Starting, restarting, and scaling

First start: expect about BootstrapGraceSeconds (30 s) before the cluster forms. Until then, GET /cluster reports enabled: false or no leader.
Restarts: a node rejoins with its previous identity and data. Restart one node at a time, and wait for GET /cluster to show it back before restarting the next.
Adding a node: start it with the same cluster settings and a new NodeName, and add it to Seeds on the other nodes if you use seeds. The leader admits it and moves some partitions to it.
Removing a node: stop it. After MemberRemovalGraceSeconds it leaves the membership and its partitions stay on the remaining nodes. In Kubernetes, a pod the headless Service no longer lists leaves after about 2 minutes (MemberAbsentFromDiscoverySeconds, 120 seconds by default), once its cluster port fails a TCP probe.
Upgrades: all nodes of a cluster must run the same broker version. To upgrade a cluster from 2.14 or earlier, stop all nodes and upgrade them together. LoT changes are refused until every node runs a version that replicates LoT.

Monitoring

Signal	What it shows
`GET /cluster`	The node’s full cluster view: leader, members, partition ownership, replication and forwarding counters
`GET /cluster/trace`	Each QoS 1 or 2 message on the node, when `ReplicationTrace` is on
`GET /info`	A short `cluster` summary: role, leader, member count
Prometheus `/metrics`	`coreflux_cluster_`, `coreflux_partition`, and `coreflux_replication_*` series
Retained `$SYS` topics	`$SYS/Coreflux/Cluster/Role`, `IsLeader`, `Leader`, `Term`, `Members`, and `Status` (a JSON summary), refreshed every 7 seconds

Watch these values; on a healthy cluster they stay at the healthy value:

Value	Healthy	When it is not
Leader (`leadership.leader` in `/cluster`)	the same node on every node	Elections in progress, or a node cut off from the others
`membersTotal`	your number of nodes, on every node	A larger value means a node’s old address is still listed; it is removed within seconds when the node answers at its new address
`membersAlive` on the leader	equal to `membersTotal`	A node is down or unreachable (followers report other nodes as `unknown`; trust the leader’s view)
Rejected publishes (`coreflux_replication_quorum_failures_total`)	not increasing	Clients are getting `0x97`: a failover is in progress, or there is no majority
Messages waiting for other nodes (`coreflux_replication_forward_backlog`)	close to `0`	A node is down or slow; the queue empties when it catches up
Messages dropped for a node that was down too long (`coreflux_replication_forward_dropped_total`)	`0`	Subscribers on that node missed messages; increase `ReplicationForwardQueueEntries` or bring nodes back sooner

Use MQTT Explorer or any MQTT client to subscribe to $SYS/Coreflux/Cluster/#.

Best practices

Run 3 or 5 nodes, and spread them across failure domains (hosts, racks, zones).
Set SharedSecret, and keep the cluster port on a private network.
Enable durability on every node.
Use MQTT 5 for QoS 1/2 publishers, and retry on 0x97.
Make consumers tolerate duplicates.
Give each node a stable NodeName and persistent storage for DataDir and the durability data.
Deploy LoT through any one node and let the cluster place it; pin an exclusive Connector to a node with CLUSTER_NODE only when the device is cabled to that node (see LoT in a cluster).
Create the same users on every node, and use the same time zone and synchronised clocks.

Don't

Don’t run 2 nodes: losing either one stops QoS 1/2 publishing, because 1 of 2 is not a majority.
Don’t set ReplicationFactor to 1 when you need messages to survive a node failure.
Don’t change PartitionCount on a running cluster.
Don’t restart several nodes at once on a 3-node cluster: QoS 1/2 publishing pauses until a majority is back.

Limitations

Offline queues stay on their node. A persistent session’s subscriptions follow a client to another node (with durability), but messages already queued for it stay on the node it was connected to, and are delivered only if it reconnects there.

Users and credentials are kept per node. Create the same users on every node. LoT definitions, including Rules, are replicated; users are not.

Retained messages are kept per node. Each node keeps its own copy of retained messages. A node added later does not receive retained messages published before it joined, and a node that was down misses retained QoS 0 messages published meanwhile.

At least once, not exactly once. After a failover, subscribers can receive some messages twice, at QoS 1 and QoS 2.

MQTT 3.1.1 publishers cannot see rejected publishes. When the cluster cannot reach a majority, a 3.1.1 client still receives a normal acknowledgement for a message that was not accepted. Use MQTT 5 for QoS 1/2 publishers.

One publishing connection is limited by the time to reach a majority. Each QoS 1/2 message waits for its majority before the broker reads the same connection’s next message. Total throughput grows with the number of publishing connections. A single connection that publishes at high rate is limited to roughly one message per round trip between nodes.

Shared subscriptions are per node. A shared subscription group includes only the clients connected to the same node; each node delivers a message to one member of its own group. See shared subscriptions.

Agents are not replicated. An agent (DEFINE AGENT) exists only on the node where you defined it. It is not copied to the other nodes, does not move when a node fails, and a -removeAgent on one node does not remove it from the others. Define an agent on every node that should run it. The LLM Connector an agent uses is an ordinary Connector and is replicated.

Agents inside a Thing. Currently:

The leader deploys a Thing (-addThing). Its definition and its connectors, models, actions, rules, and panels replicate to every node.
A DEFINE AGENT inside the Thing is not replicated. It currently runs only on the node that was leader when the Thing was deployed, which is not necessarily the node that received the command.
The agent’s conversation memory stays on that node and is never replicated.
The agent does not move on failover. If its node is down, the agent is unavailable until that node returns.
On the other nodes the Thing still lists the agent, but the agent is not running there, so Thing health on those nodes may show it as not ok. Its status topics are visible on every node.

Next Steps

MQTT State Durability

Keep sessions, queued messages, and retained messages across restarts on every node.

Operational Endpoints

Monitor each node with /cluster, /info, and Prometheus metrics.

​Keep messaging available when a node fails

​When to use clustering

​Quick Start

​How a cluster works

​One leader, automatic membership

​Partitions spread the work

​QoS 1 and 2 are acknowledged once they are safe

​Subscribers on any node receive every message

​Failover

​Sessions follow the client

​Durability and clustering

​LoT in a cluster

​The three rules

​Where LoT runs

​Where connectors run

​Pin a connector

​Change definitions

​Owners and failover

​ON START and KEEP

​Server connectors

​What is guaranteed

​Configuration reference

​Identity and discovery

​Network

​Replication

​Membership and startup

​Ports and network

​Delivery guarantees

​Operations

​Starting, restarting, and scaling

​Monitoring

​Best practices

​Limitations

​Next Steps

MQTT State Durability

Operational Endpoints

Keep messaging available when a node fails

When to use clustering

Quick Start

How a cluster works

One leader, automatic membership

Partitions spread the work

QoS 1 and 2 are acknowledged once they are safe

Subscribers on any node receive every message

Failover

Sessions follow the client

Durability and clustering

LoT in a cluster

The three rules

Where LoT runs

Where connectors run

Pin a connector

Change definitions

Owners and failover

ON START and KEEP

Server connectors

What is guaranteed

Configuration reference

Identity and discovery

Network

Replication

Membership and startup

Ports and network

Delivery guarantees

Operations

Starting, restarting, and scaling

Monitoring

Best practices

Limitations

Next Steps