> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreflux.org/llms.txt > Use this file to discover all available pages before exploring further. # Store and Forward Configuration > How LoT connectors buffer MQTT while a destination is slow, bound that buffer with ADD ROUTE_CONFIG, and choose backpressure or drop, plus what this does not cover ## Store and forward in LoT connectors Store and forward means holding messages while the next hop is slow or unavailable, then passing them on when it catches up. In Coreflux, every LoT connector sits behind a shared, **bounded, in-memory queue** between the broker and the connector's handlers. `ADD ROUTE_CONFIG` decides what each connector does when that queue is full: wait (backpressure) or drop the newest message. > [!TIP] > > **Like a ticket window with a limited line.** People already in line are always served, in order. When the line is full you can wait for a spot, wait a short time then turn the newest arrival away, or turn them away immediately. ### The principles, and how LoT handles them | Principle | In LoT connectors | | - | - | | **Buffer while the destination is slow** | Matching messages wait in a shared in-memory queue that one consumer drains in order. | | **Bounded queues** | The shared queue holds `COREFLUX_ROUTE_CHANNEL_CAPACITY` messages (default `100000`). `OVERLOAD_CAPACITY` puts a private cap on one connector. | | **Backpressure or drop** | `lossless` makes the publisher wait. `balanced` waits up to `OVERLOAD_TIMEOUT`, then drops. `realtime` drops at once. Only the **arriving** message is ever dropped. | | **Ordering** | First in, first out. An accepted message is never evicted or reordered, so a sequence such as Sparkplug birth then data stays consistent. | | **Delivery guarantee** | At most once into the connector. `lossless` means "never dropped by this queue", not "delivered to the destination". | | **Survives a restart** | No. The queue lives in memory. Anything still queued when the broker stops is lost. | ### What it does not do > [!WARNING] > > This is not a complete store-and-forward system. The connector queue: > > * **does not write to disk**, so queued messages don't survive a broker restart or crash, > * **does not replay** messages after a destination comes back. Once a connector has taken a message from the queue, retrying a failed write is up to that connector type, and many simply log the failure, > * **does not acknowledge end to end**: a publisher's QoS 1/2 acknowledgement means the broker accepted the message, not that the destination stored it. ### Filling the gaps * **Upstream, use MQTT itself.** Publish with QoS 1 or 2 from clients with persistent sessions (`cleanSession=false`, or a session expiry in MQTT 5), so the broker keeps their messages while they are offline. The broker's [durability](/latest/mqtt-broker/durability) settings decide what survives a broker restart. * **Store first, forward later.** Write each message to a database connector (or a model with `STORE IN`), then forward from that table on a schedule. The table is the durable buffer and can be replayed. * **Let the destination buffer.** Kafka, and a remote MQTT broker with persistent sessions, keep their own durable queues once a message reaches them. * **Watch for drops.** The broker logs when a connector starts dropping. Treat that as an alert. ### When to use it * Leave the default (`balanced`) on most connectors. * Use `lossless` when a path must not lose messages (audit, compliance) and you accept that the publisher may wait. * Use `realtime` when high-rate telemetry must never stall the client that published it. * Set `OVERLOAD_CAPACITY` to cap one noisy connector so it cannot crowd out others. `ADD ROUTE_CONFIG` is the only LoT block that works on **every** connector type. Place it next to that type's own config (`SQL_CONFIG`, `SOURCE_CONFIG`, `MODBUS_CONFIG`, and so on). Order among `ADD` blocks does not matter. *** Let's look at how to set it, starting with the three common contracts. ## Quick Start **Never drop a message** Use `lossless` on a forwarding path that must keep every message. The publisher waits until this connector has room — it never drops. ```lot wrap focus={3-4} theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}} DEFINE ROUTE CloudBridge WITH TYPE MQTT_BRIDGE ADD ROUTE_CONFIG WITH OVERLOAD_POLICY "lossless" ADD SOURCE_CONFIG WITH BROKER SELF ADD DESTINATION_CONFIG WITH BROKER_ADDRESS "broker.example.com" WITH BROKER_PORT '8883' WITH USE_TLS "true" ADD MAPPING auditForward WITH SOURCE_TOPIC "audit/#" WITH DESTINATION_TOPIC "cloud/audit/#" WITH DIRECTION "out" ``` **Never stall the publisher** Use `realtime` on high-rate telemetry. If the connector is full, the arriving message is dropped immediately and the publishing client is not delayed. ```lot wrap focus={3-4} theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}} DEFINE ROUTE SensorDB WITH TYPE POSTGRESQL ADD ROUTE_CONFIG WITH OVERLOAD_POLICY "realtime" ADD SQL_CONFIG WITH SERVER GET ENV "PG_HOST" WITH DATABASE "factory" WITH USERNAME GET ENV "PG_USER" WITH PASSWORD GET SECRET "PG_PASSWORD" ADD EVENT StoreReading WITH SOURCE_TOPIC "sensors/+/reading" WITH QUERY "INSERT INTO readings (ts, value) VALUES (NOW(), '{value.json}')" ``` **Cap a noisy connector** Cap how many of **this connector's** messages may be in flight. Extra messages from this connector drop even while other connectors still have room. ```lot wrap focus={3-4} theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}} DEFINE ROUTE NoisySensors WITH TYPE MQTT_BRIDGE ADD ROUTE_CONFIG WITH OVERLOAD_CAPACITY 5000 ADD SOURCE_CONFIG WITH BROKER SELF ADD DESTINATION_CONFIG WITH BROKER_ADDRESS "telemetry.example.com" WITH BROKER_PORT '1883' ADD MAPPING sensorsOut WITH SOURCE_TOPIC "sensors/#" WITH DESTINATION_TOPIC "remote/sensors/#" WITH DIRECTION "out" ``` *** ## Settings All three settings are optional. An absent `ADD ROUTE_CONFIG` block inherits the broker default entirely. ```lot wrap focus={3-6} theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}} DEFINE ROUTE CloudBridge WITH TYPE MQTT_BRIDGE ADD ROUTE_CONFIG WITH OVERLOAD_POLICY "balanced" WITH OVERLOAD_TIMEOUT 200 WITH OVERLOAD_CAPACITY 20000 ADD SOURCE_CONFIG WITH BROKER SELF ADD DESTINATION_CONFIG WITH BROKER_ADDRESS "broker.example.com" WITH BROKER_PORT '8883' WITH USE_TLS "true" ADD MAPPING sensorSync WITH SOURCE_TOPIC "sensors/#" WITH DESTINATION_TOPIC "cloud/sensors/#" WITH DIRECTION "out" ``` | Setting | Required | Default | Description | | - | - | - | - | | `OVERLOAD_POLICY` | No | `balanced` | `lossless` waits until there is room and never drops. `balanced` waits up to `OVERLOAD_TIMEOUT`, then drops the **arriving** message. `realtime` never waits — it drops immediately when the connector is full. | | `OVERLOAD_TIMEOUT` | No | `50` (ms) | Wait budget for `balanced` only. Ignored for `lossless` (waits indefinitely) and `realtime` (never waits). Must be zero or greater. | | `OVERLOAD_CAPACITY` | No | unbounded | Maximum number of **this connector's** messages allowed in flight at once. Once reached, further messages from this connector drop even if other connectors still have room. Must be greater than zero. | A message already accepted for a connector is never evicted. Only the arriving message can be dropped. That keeps sequences such as Sparkplug birth then data self-consistent under load. > [!WARNING] > > Do not set `OVERLOAD_CAPACITY` together with `OVERLOAD_POLICY "lossless"`. A capacity cap drops overflow; `lossless` never drops. The broker logs an error naming the connector, **ignores the capacity**, and runs the connector as lossless. The same conflict applies if the connector inherits `lossless` from the broker default and then declares a capacity. *** ## Defaults Each setting resolves on its own: 1. **No `ADD ROUTE_CONFIG` block** — the connector inherits the broker default for policy, timeout, and capacity. 2. **A block is present but a setting is omitted** — that setting does not copy the broker's timeout or capacity. `OVERLOAD_POLICY` inherits the broker default. `OVERLOAD_TIMEOUT` and `OVERLOAD_CAPACITY` use that policy's own default (`50` ms for `balanced`, unbounded capacity). 3. **An invalid value** (unknown policy name, negative timeout, non-positive capacity) does not stop the connector loading. The broker logs an error and uses that field's default. Broker-wide defaults are environment variables, read once at process start. Change them and **restart** the broker. A connector's own `ADD ROUTE_CONFIG` always overrides these for that connector. | Variable | Default | Effect | | - | - | - | | `COREFLUX_ROUTE_OVERLOAD_POLICY` | `balanced` | Default for `OVERLOAD_POLICY` (`lossless`, `balanced`, or `realtime`). An unrecognized value logs an error and falls back to `balanced`. | | `COREFLUX_ROUTE_CHANNEL_FULL_TIMEOUT_MS` | `50` | Default wait budget in milliseconds for `balanced`. Ignored when the resolved policy is `lossless` or `realtime`. A negative value logs an error and falls back to `50`. | | `COREFLUX_ROUTE_CHANNEL_CAPACITY` | `100000` | Shared ceiling on how many connector updates the broker will hold at once. Below the ceiling, matching messages are accepted. At the ceiling, the arriving message's wait-or-drop policy applies. Values below `1024` are raised to `1024`. Unset, non-numeric, zero, or negative values fall back to `100000`. | `OVERLOAD_CAPACITY` on a connector is a **private** budget for that connector. The shared ceiling is separate: one connector can hit its own cap and start dropping while other connectors still have room. See [Broker Configuration](/latest/mqtt-broker/configuration#performance-and-back-pressure) for how these sit alongside other broker limits. *** ## What happens under load When a connector is waiting, that wait can delay **the publishing client's own publish** — only that client, and only up to the wait budget. Other connected clients are not stalled by this connector. The broker logs two different saturation conditions (each at most once every five seconds): * The shared connector queue is full — every connector waiting behind it is affected. * A single connector is over the `OVERLOAD_CAPACITY` it declared for itself — the shared queue may still have room. > [!WARNING] > > If two connectors could match the same topic (for example one exact mapping and another `sensors/#`), the broker applies the **stricter** wait-or-drop choice among those candidates. A `lossless` connector on an overlapping pattern can make a `realtime` connector wait, even if only the `realtime` connector would have handled the message. The broker never drops a message because a more lenient connector asked to drop it. If mixed policies on overlapping wildcards wait or drop more than you expect, check the patterns first — that behavior is by design. *** ## Troubleshooting > **A connector drops messages even though the broker looks idle** > > That connector's own `OVERLOAD_CAPACITY` is exhausted. Look for the broker warning that a connector is over its in-flight budget, not the warning that the shared queue is full. > **Publishers on a lossless connector stall under load** > > Working as designed — `lossless` waits until there is room. Switch to `balanced` with a timeout if you would rather drop than stall, or cap a noisy connector with `OVERLOAD_CAPACITY` so it drops its own overflow instead. > **OVERLOAD\_CAPACITY seems to have no effect** > > It is declared alongside `OVERLOAD_POLICY "lossless"` (or the broker default is `lossless`). Check the broker log for the lossless/capacity conflict — capacity is ignored, not applied. > **A connector with no wildcard still waits longer than its own policy** > > Another connector's overlapping wildcard pattern resolved a stricter policy for that topic. See the overlapping-pattern warning above. > **The connector drops immediately and never waits** > > The resolved policy is `realtime`. That is expected — `realtime` drops without waiting. *** ## Best practices * Start with the default `balanced` policy. Change a connector only when you have a reason (must not lose, must not stall, or must not crowd others). * Use `lossless` sparingly. It protects delivery at the cost of delaying the publisher that caused the backlog. * Put `OVERLOAD_CAPACITY` on the noisy connector, not on the critical one — the critical connector can stay unbounded while the noisy connector drops its own overflow. * Prefer exact topic mappings when two connectors would otherwise share a broad `+` / `#` pattern with different policies. * Invalid settings never fail the connector silently: read the broker log after deploy if behavior does not match the block you wrote. *** ## Next Steps > **[Connectors Overview](./overview)** > > Define connectors, mappings, and status topics. > **[Broker Configuration](/latest/mqtt-broker/configuration#performance-and-back-pressure)** > > Set broker-wide queue defaults and other limits. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.