> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreflux.org/llms.txt
> Use this file to discover all available pages before exploring further.

# High-Availability Clustering

> Run multiple Coreflux MQTT Broker nodes as a fault-tolerant cluster with leader election, durable QoS replication, and automatic failover

## Keep messaging available when a node fails

A single broker is a single point of failure. High-availability clustering runs several Coreflux MQTT Broker nodes as one system: if one node stops, the others keep serving clients. Clients can connect to any healthy node. Important messages (QoS 1 and 2) are copied across nodes so a crash of a minority of the cluster cannot erase a message the broker already accepted.

> [!TIP]
>
> **Like a team with a captain and shared playbook.** One node leads changes; the others stay ready. If the captain drops out, the team elects a new one and keeps playing — without rewriting every client's connection plan.

### When to use clustering

Use clustering when you need:

* continuous MQTT service if one broker stops or is restarted;
* important publishes (QoS 1/2) to survive if some of the nodes crash after the broker has accepted them; or
* a fixed or elastic set of broker nodes that should act as one cluster.

Prefer **odd** node counts (**3** or **5**) so a clear majority can decide who leads. Clustering requires a license that grants the cluster feature. If you enable clustering without that license, the broker logs a warning and runs standalone.

> [!NOTE]
>
> Clustering is **broker configuration**, not a LoT System Route. For topic sync with another MQTT broker (including non-Coreflux), use an [MQTT Bridge](/v2.14/lot-language/routes/system/mqtt-bridge) instead.

***

Configure each node with the same cluster identity and discovery settings, then deploy the same LoT project to every node through your normal workflow.

## Quick Start

Add a `Cluster` section to the broker configuration (or matching environment variables — env vars override the file). On **every** node, create a dedicated MQTT user for inter-node traffic and set `DataPlaneUsername` to that username so nodes can exchange topic data.

**Seeds (typical production)**

Use fixed peer addresses when automatic discovery is unavailable (cloud, most Kubernetes networks, multi-subnet LANs):

```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "factory-floor",
    "NodeName": "broker-a",
    "Discovery": "seeds",
    "Seeds": ["broker-a:9200", "broker-b:9200", "broker-c:9200"],
    "SharedSecret": "a-strong-secret",
    "DataPlaneUsername": "cluster"
  }
}
```

**Multicast (same LAN)**

Auto-discover peers on a shared local network (default discovery mode):

```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "lab-cluster",
    "NodeName": "broker-1",
    "Discovery": "multicast",
    "SharedSecret": "a-strong-secret",
    "DataPlaneUsername": "cluster"
  }
}
```

**Kubernetes**

Discover peers via a headless Service DNS name; better for elastic pods than a static address list:

```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{
  "Cluster": {
    "Enabled": true,
    "ClusterId": "k8s-brokers",
    "NodeName": "broker-0",
    "Discovery": "kubernetes",
    "KubernetesServiceDns": "coreflux-broker-headless.default.svc.cluster.local",
    "KubernetesPeerPort": 9200,
    "SharedSecret": "a-strong-secret",
    "DataPlaneUsername": "cluster"
  }
}
```

***

## Concepts operators need

| Plane | What it does for you |
| - | - |
| **Control plane** | How nodes find each other, who leads, and which node may accept LoT **writes** |
| **Data plane** | Moves MQTT topic traffic between nodes so subscribers on any node can receive publishes |

* **One leader.** Runtime LoT **write** commands run only on the current leader. Followers reject writes and tell you which node is the leader. Read-only and node-local commands (queries, diagnostics, license/instance key) still run locally.
* **LoT is not copied between nodes.** Each node loads its own LoT from disk / GitOps. A write applied on the leader changes **only that node** until you distribute the same project files to the others. Point runtime tooling at the leader, and treat your project repository as the source of truth for all nodes.
* **Only matching nodes join.** Nodes must share the same cluster identity and join secret so unrelated brokers do not join by accident. (See `ClusterId` and `SharedSecret` in the configuration reference.)
* **Session failover needs durability.** Enable [MQTT state durability](/v2.14/mqtt-broker/durability) so a client that reconnects to another node can get its subscriptions back. Messages queued for offline clients stay on the node that hosted them (see [Limitations](#limitations)).

***

## Configuration reference

### Core options

| Config field | Env var | Default | Description |
| - | - | - | - |
| `Enabled` | `COREFLUX_CLUSTER_ENABLED` | `false` | Master switch |
| `ClusterId` | `COREFLUX_CLUSTER_ID` | `coreflux-cluster` | Logical cluster identity |
| `NodeName` | `COREFLUX_CLUSTER_NODE_NAME` | hostname | This node's identity |
| `BindIP` | `COREFLUX_CLUSTER_BIND_IP` | `0.0.0.0` | Control-plane listener bind address |
| `Port` | `COREFLUX_CLUSTER_PORT` | `9200` | Control-plane TCP port |
| `AdvertisedHost` | `COREFLUX_CLUSTER_ADVERTISED_HOST` | `NodeName` | Address peers use to reach this node |
| `AdvertisedPort` | `COREFLUX_CLUSTER_ADVERTISED_PORT` | `Port` | Port peers use to reach this node |
| `Discovery` | `COREFLUX_CLUSTER_DISCOVERY` | `multicast` | `multicast`, `seeds`, or `kubernetes` |
| `Seeds` | `COREFLUX_CLUSTER_SEEDS` | `[]` | Comma/semicolon list of `host:port` |
| `MulticastGroup` | `COREFLUX_CLUSTER_MULTICAST_GROUP` | `239.63.114.71` | Multicast group |
| `MulticastPort` | `COREFLUX_CLUSTER_MULTICAST_PORT` | `9201` | Multicast announce port |
| `KubernetesServiceDns` | `COREFLUX_CLUSTER_K8S_SERVICE_DNS` | — | Headless Service DNS name |
| `KubernetesPeerPort` | `COREFLUX_CLUSTER_K8S_PEER_PORT` | — | Peer port for Kubernetes discovery |
| `SharedSecret` | `COREFLUX_CLUSTER_SHARED_SECRET` | — | Join secret (discovery filtering and data-plane integrity) |
| `DataPlaneEnabled` | `COREFLUX_CLUSTER_DATAPLANE_ENABLED` | `true` | Exchange MQTT traffic between nodes |
| `DataPlaneMqttPort` | `COREFLUX_CLUSTER_DATAPLANE_MQTT_PORT` | broker MQTT port | MQTT port used between nodes |
| `DataPlaneUsername` | `COREFLUX_CLUSTER_DATAPLANE_USERNAME` | — | MQTT username for inter-node links (password from the user store) |
| `UseTls` | `COREFLUX_CLUSTER_TLS` | `false` | TLS on inter-node MQTT links |
| `Topology` | `COREFLUX_CLUSTER_TOPOLOGY` | `mesh` | `mesh` (recommended) or `ring` (complete only for ≤ 3 nodes) |
| `DataDir` | `COREFLUX_CLUSTER_DATA_DIR` | `{CFBasePath}/cluster` | Cluster state directory |

The data plane starts only when `DataPlaneEnabled` is true **and** `DataPlaneUsername` names a user that exists in the broker MQTT user store on that node.

### Replication options

| Config field | Env var | Default | Description |
| - | - | - | - |
| `ReplicationEnabled` | `COREFLUX_CLUSTER_REPLICATION_ENABLED` | `true` | Require quorum durability before acknowledging QoS ≥ 1 publishes |
| `ReplicationFactor` | `COREFLUX_CLUSTER_REPLICATION_FACTOR` | `0` (= auto min(3, N)) | Replica-set size per partition |
| `PartitionCount` | `COREFLUX_CLUSTER_PARTITION_COUNT` | `32` | Logical partitions (pick once; changing re-hashes keys) |
| `ReplicationTimeoutMs` | `COREFLUX_CLUSTER_REPLICATION_TIMEOUT_MS` | `5000` | Quorum wait before rejecting a publish |
| `ReplicationReplaySeconds` | `COREFLUX_CLUSTER_REPLICATION_REPLAY_SECONDS` | `300` | Failover replay window (0 = no replay) |

***

## Ports

| Port | Protocol | Purpose |
| - | - | - |
| `9200` | TCP | Control plane (leader election and membership) |
| `9201` | UDP | Multicast discovery (multicast mode only) |
| Broker MQTT port | TCP | Inter-node MQTT data plane |

Open these between cluster members in your firewall or network policies.

***

## Delivery guarantees

| QoS | What clustering adds |
| - | - |
| **0** | Best-effort delivery to peers; may be lost if a target node is down |
| **1** | The broker accepts the publish only after a **majority** of the replica set has stored it; other nodes receive it reliably; after failover it may be delivered again (at-least-once) |
| **2** | Same majority check before the broker accepts the publish in normal operation; across failover, treat delivery as **at-least-once** and design consumers to ignore duplicates |

**Summary:** once the broker has accepted a QoS 1 or 2 publish, that message is stored on a majority of nodes and can be replayed if ownership moves after a failure. If the cluster is split and this side no longer has a majority, it **rejects** those publishes instead of accepting messages it cannot keep safe.

***

## Operations

### Bootstrap and failover

* A node that sees an existing leader joins as a follower.
* If no leader is visible and peers exist, the cluster elects a leader and adds members as they are discovered.
* When the leader is lost, a new one is elected quickly (typically within a fraction of a second). Survivors stop contacting confirmed-dead peers and bring returning nodes back into the mesh automatically.
* With **seeds** discovery, a stopped node can remain a configured member (marked unavailable) while majority is computed over the configured list. Prefer **kubernetes** discovery when instances come and go often.

### Observability

| Signal | What it shows |
| - | - |
| `GET /cluster` | Full status: role, leadership, members, data-plane health, replication, and related fields |
| `GET /info` | Lightweight `cluster` summary |
| Prometheus `/metrics` | Cluster, partition, replication, and related series (`coreflux_cluster_*`, `coreflux_replication_*`, and related) |
| Retained `$SYS` topics | `$SYS/Coreflux/Cluster/{Role,IsLeader,Leader,Term,Members,Status}` (refreshed periodically) |

Use MQTT Explorer or any MQTT client to subscribe to the `$SYS/Coreflux/Cluster/#` topics, or call `GET /cluster` on each node's operations HTTP port.

***

## Limitations

> [!WARNING]
>
> **LoT is not replicated between nodes.** Distribute the same project/configuration to every node. Runtime edits on the leader do not appear on followers until you provision them.

> [!WARNING]
>
> **Retained messages are node-local.** They survive a restart on that node but are not a cluster-wide retained store. A subscriber on another node may not see a retain set elsewhere.

> [!WARNING]
>
> **Subscriptions can move; offline queues do not.** A client reconnecting with a persistent session to another node can regain its subscriptions when [durability](/v2.14/mqtt-broker/durability) is enabled. Queued QoS 1/2 messages for offline clients stay on the node that hosted them until that node returns.

> [!WARNING]
>
> **QoS 2 across failover is at-least-once.** Design consumers to tolerate duplicates after a node failure or replay.

***

## Next Steps

> **[MQTT State Durability](/v2.14/mqtt-broker/durability)**
>
> Persist retained messages, sessions, and queued QoS across restarts — required for session failover.

> **[Broker Configuration](/v2.14/mqtt-broker/configuration)**
>
> Ports, TLS, listeners, and other broker settings for each node.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.