> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreflux.org/llms.txt
> Use this file to discover all available pages before exploring further.

# High-Availability Clustering

> Run multiple Coreflux MQTT Broker nodes as a fault-tolerant cluster with leader election, durable QoS replication, and automatic failover

## Keep messaging available when a node fails

A single broker is a single point of failure. High-availability clustering runs several Coreflux MQTT Broker nodes as one system: if one node stops, the others keep serving clients. Clients can connect to any healthy node. Important messages (QoS 1 and 2) are copied across nodes so a crash of a minority of the cluster cannot erase a message the broker already accepted.

<Tip>
  **Like a team with a captain and shared playbook.** One node leads changes; the others stay ready. If the captain drops out, the team elects a new one and keeps playing — without rewriting every client's connection plan.
</Tip>

### When to use clustering

Use clustering when you need:

* continuous MQTT service if one broker stops or is restarted;
* important publishes (QoS 1/2) to survive if some of the nodes crash after the broker has accepted them; or
* a fixed or elastic set of broker nodes that should act as one cluster.

Prefer **odd** node counts (**3** or **5**) so a clear majority can decide who leads. Clustering requires a license that grants the cluster feature. If you enable clustering without that license, the broker logs a warning and runs standalone.

<Note>
  Clustering is **broker configuration**, not a LoT System Route. For topic sync with another MQTT broker (including non-Coreflux), use an [MQTT Bridge](/latest/lot-language/routes/system/mqtt-bridge) instead.
</Note>

***

Configure each node with the same cluster identity and discovery settings, then deploy the same LoT project to every node through your normal workflow.

## Quick Start

Add a `Cluster` section to the broker configuration (or matching environment variables — env vars override the file). On **every** node, create a dedicated MQTT user for inter-node traffic and set `DataPlaneUsername` to that username so nodes can exchange topic data.

<Tabs>
  <Tab title="Seeds (typical production)">
    Use fixed peer addresses when automatic discovery is unavailable (cloud, most Kubernetes networks, multi-subnet LANs):

    ```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
    {
      "Cluster": {
        "Enabled": true,
        "ClusterId": "factory-floor",
        "NodeName": "broker-a",
        "Discovery": "seeds",
        "Seeds": ["broker-a:9200", "broker-b:9200", "broker-c:9200"],
        "SharedSecret": "a-strong-secret",
        "DataPlaneUsername": "cluster"
      }
    }
    ```
  </Tab>

  <Tab title="Multicast (same LAN)">
    Auto-discover peers on a shared local network (default discovery mode):

    ```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
    {
      "Cluster": {
        "Enabled": true,
        "ClusterId": "lab-cluster",
        "NodeName": "broker-1",
        "Discovery": "multicast",
        "SharedSecret": "a-strong-secret",
        "DataPlaneUsername": "cluster"
      }
    }
    ```
  </Tab>

  <Tab title="Kubernetes">
    Discover peers via a headless Service DNS name; better for elastic pods than a static address list:

    ```json wrap theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
    {
      "Cluster": {
        "Enabled": true,
        "ClusterId": "k8s-brokers",
        "NodeName": "broker-0",
        "Discovery": "kubernetes",
        "KubernetesServiceDns": "coreflux-broker-headless.default.svc.cluster.local",
        "KubernetesPeerPort": 9200,
        "SharedSecret": "a-strong-secret",
        "DataPlaneUsername": "cluster"
      }
    }
    ```
  </Tab>
</Tabs>

***

## Concepts operators need

| Plane             | What it does for you                                                                    |
| ----------------- | --------------------------------------------------------------------------------------- |
| **Control plane** | How nodes find each other, who leads, and which node may accept LoT **writes**          |
| **Data plane**    | Moves MQTT topic traffic between nodes so subscribers on any node can receive publishes |

* **One leader.** Runtime LoT **write** commands run only on the current leader. Followers reject writes and tell you which node is the leader. Read-only and node-local commands (queries, diagnostics, license/instance key) still run locally.
* **LoT is not copied between nodes.** Each node loads its own LoT from disk / GitOps. A write applied on the leader changes **only that node** until you distribute the same project files to the others. Point runtime tooling at the leader, and treat your project repository as the source of truth for all nodes.
* **Only matching nodes join.** Nodes must share the same cluster identity and join secret so unrelated brokers do not join by accident. (See `ClusterId` and `SharedSecret` in the configuration reference.)
* **Session failover needs durability.** Enable [MQTT state durability](/latest/mqtt-broker/durability) so a client that reconnects to another node can get its subscriptions back. Messages queued for offline clients stay on the node that hosted them (see [Limitations](#limitations)).

***

## Configuration reference

### Core options

| Config field           | Env var                                | Default                | Description                                                       |
| ---------------------- | -------------------------------------- | ---------------------- | ----------------------------------------------------------------- |
| `Enabled`              | `COREFLUX_CLUSTER_ENABLED`             | `false`                | Master switch                                                     |
| `ClusterId`            | `COREFLUX_CLUSTER_ID`                  | `coreflux-cluster`     | Logical cluster identity                                          |
| `NodeName`             | `COREFLUX_CLUSTER_NODE_NAME`           | hostname               | This node's identity                                              |
| `BindIP`               | `COREFLUX_CLUSTER_BIND_IP`             | `0.0.0.0`              | Control-plane listener bind address                               |
| `Port`                 | `COREFLUX_CLUSTER_PORT`                | `9200`                 | Control-plane TCP port                                            |
| `AdvertisedHost`       | `COREFLUX_CLUSTER_ADVERTISED_HOST`     | `NodeName`             | Address peers use to reach this node                              |
| `AdvertisedPort`       | `COREFLUX_CLUSTER_ADVERTISED_PORT`     | `Port`                 | Port peers use to reach this node                                 |
| `Discovery`            | `COREFLUX_CLUSTER_DISCOVERY`           | `multicast`            | `multicast`, `seeds`, or `kubernetes`                             |
| `Seeds`                | `COREFLUX_CLUSTER_SEEDS`               | `[]`                   | Comma/semicolon list of `host:port`                               |
| `MulticastGroup`       | `COREFLUX_CLUSTER_MULTICAST_GROUP`     | `239.63.114.71`        | Multicast group                                                   |
| `MulticastPort`        | `COREFLUX_CLUSTER_MULTICAST_PORT`      | `9201`                 | Multicast announce port                                           |
| `KubernetesServiceDns` | `COREFLUX_CLUSTER_K8S_SERVICE_DNS`     | —                      | Headless Service DNS name                                         |
| `KubernetesPeerPort`   | `COREFLUX_CLUSTER_K8S_PEER_PORT`       | —                      | Peer port for Kubernetes discovery                                |
| `SharedSecret`         | `COREFLUX_CLUSTER_SHARED_SECRET`       | —                      | Join secret (discovery filtering and data-plane integrity)        |
| `DataPlaneEnabled`     | `COREFLUX_CLUSTER_DATAPLANE_ENABLED`   | `true`                 | Exchange MQTT traffic between nodes                               |
| `DataPlaneMqttPort`    | `COREFLUX_CLUSTER_DATAPLANE_MQTT_PORT` | broker MQTT port       | MQTT port used between nodes                                      |
| `DataPlaneUsername`    | `COREFLUX_CLUSTER_DATAPLANE_USERNAME`  | —                      | MQTT username for inter-node links (password from the user store) |
| `UseTls`               | `COREFLUX_CLUSTER_TLS`                 | `false`                | TLS on inter-node MQTT links                                      |
| `Topology`             | `COREFLUX_CLUSTER_TOPOLOGY`            | `mesh`                 | `mesh` (recommended) or `ring` (complete only for ≤ 3 nodes)      |
| `DataDir`              | `COREFLUX_CLUSTER_DATA_DIR`            | `{CFBasePath}/cluster` | Cluster state directory                                           |

The data plane starts only when `DataPlaneEnabled` is true **and** `DataPlaneUsername` names a user that exists in the broker MQTT user store on that node.

### Replication options

| Config field               | Env var                                       | Default                | Description                                                      |
| -------------------------- | --------------------------------------------- | ---------------------- | ---------------------------------------------------------------- |
| `ReplicationEnabled`       | `COREFLUX_CLUSTER_REPLICATION_ENABLED`        | `true`                 | Require quorum durability before acknowledging QoS ≥ 1 publishes |
| `ReplicationFactor`        | `COREFLUX_CLUSTER_REPLICATION_FACTOR`         | `0` (= auto min(3, N)) | Replica-set size per partition                                   |
| `PartitionCount`           | `COREFLUX_CLUSTER_PARTITION_COUNT`            | `32`                   | Logical partitions (pick once; changing re-hashes keys)          |
| `ReplicationTimeoutMs`     | `COREFLUX_CLUSTER_REPLICATION_TIMEOUT_MS`     | `5000`                 | Quorum wait before rejecting a publish                           |
| `ReplicationReplaySeconds` | `COREFLUX_CLUSTER_REPLICATION_REPLAY_SECONDS` | `300`                  | Failover replay window (0 = no replay)                           |

***

## Ports

| Port             | Protocol | Purpose                                        |
| ---------------- | -------- | ---------------------------------------------- |
| `9200`           | TCP      | Control plane (leader election and membership) |
| `9201`           | UDP      | Multicast discovery (multicast mode only)      |
| Broker MQTT port | TCP      | Inter-node MQTT data plane                     |

Open these between cluster members in your firewall or network policies.

***

## Delivery guarantees

| QoS   | What clustering adds                                                                                                                                                                 |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **0** | Best-effort delivery to peers; may be lost if a target node is down                                                                                                                  |
| **1** | The broker accepts the publish only after a **majority** of the replica set has stored it; other nodes receive it reliably; after failover it may be delivered again (at-least-once) |
| **2** | Same majority check before the broker accepts the publish in normal operation; across failover, treat delivery as **at-least-once** and design consumers to ignore duplicates        |

**Summary:** once the broker has accepted a QoS 1 or 2 publish, that message is stored on a majority of nodes and can be replayed if ownership moves after a failure. If the cluster is split and this side no longer has a majority, it **rejects** those publishes instead of accepting messages it cannot keep safe.

***

## Operations

### Bootstrap and failover

* A node that sees an existing leader joins as a follower.
* If no leader is visible and peers exist, the cluster elects a leader and adds members as they are discovered.
* When the leader is lost, a new one is elected quickly (typically within a fraction of a second). Survivors stop contacting confirmed-dead peers and bring returning nodes back into the mesh automatically.
* With **seeds** discovery, a stopped node can remain a configured member (marked unavailable) while majority is computed over the configured list. Prefer **kubernetes** discovery when instances come and go often.

### Observability

| Signal                 | What it shows                                                                                                     |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `GET /cluster`         | Full status: role, leadership, members, data-plane health, replication, and related fields                        |
| `GET /info`            | Lightweight `cluster` summary                                                                                     |
| Prometheus `/metrics`  | Cluster, partition, replication, and related series (`coreflux_cluster_*`, `coreflux_replication_*`, and related) |
| Retained `$SYS` topics | `$SYS/Coreflux/Cluster/{Role,IsLeader,Leader,Term,Members,Status}` (refreshed periodically)                       |

Use MQTT Explorer or any MQTT client to subscribe to the `$SYS/Coreflux/Cluster/#` topics, or call `GET /cluster` on each node's operations HTTP port.

***

## Limitations

<Warning>
  **LoT is not replicated between nodes.** Distribute the same project/configuration to every node. Runtime edits on the leader do not appear on followers until you provision them.
</Warning>

<Warning>
  **Retained messages are node-local.** They survive a restart on that node but are not a cluster-wide retained store. A subscriber on another node may not see a retain set elsewhere.
</Warning>

<Warning>
  **Subscriptions can move; offline queues do not.** A client reconnecting with a persistent session to another node can regain its subscriptions when [durability](/latest/mqtt-broker/durability) is enabled. Queued QoS 1/2 messages for offline clients stay on the node that hosted them until that node returns.
</Warning>

<Warning>
  **QoS 2 across failover is at-least-once.** Design consumers to tolerate duplicates after a node failure or replay.
</Warning>

***

## Next Steps

<CardGroup cols={2}>
  <Card title="MQTT State Durability" icon="hard-drive" href="/latest/mqtt-broker/durability">
    Persist retained messages, sessions, and queued QoS across restarts — required for session failover.
  </Card>

  <Card title="Broker Configuration" icon="gear" href="/latest/mqtt-broker/configuration">
    Ports, TLS, listeners, and other broker settings for each node.
  </Card>
</CardGroup>