Store and Forward Configuration

Store and forward in LoT connectors

Store and forward means holding messages while the next hop is slow or unavailable, then passing them on when it catches up. In Coreflux, every LoT connector sits behind a shared, bounded, in-memory queue between the broker and the connector’s handlers. ADD ROUTE_CONFIG decides what each connector does when that queue is full: wait (backpressure) or drop the newest message.

Like a ticket window with a limited line. People already in line are always served, in order. When the line is full you can wait for a spot, wait a short time then turn the newest arrival away, or turn them away immediately.

The principles, and how LoT handles them

Principle	In LoT connectors
Buffer while the destination is slow	Matching messages wait in a shared in-memory queue that one consumer drains in order.
Bounded queues	The shared queue holds `COREFLUX_ROUTE_CHANNEL_CAPACITY` messages (default `100000`). `OVERLOAD_CAPACITY` puts a private cap on one connector.
Backpressure or drop	`lossless` makes the publisher wait. `balanced` waits up to `OVERLOAD_TIMEOUT`, then drops. `realtime` drops at once. Only the arriving message is ever dropped.
Ordering	First in, first out. An accepted message is never evicted or reordered, so a sequence such as Sparkplug birth then data stays consistent.
Delivery guarantee	At most once into the connector. `lossless` means “never dropped by this queue”, not “delivered to the destination”.
Survives a restart	No. The queue lives in memory. Anything still queued when the broker stops is lost.

What it does not do

This is not a complete store-and-forward system. The connector queue:

does not write to disk, so queued messages don’t survive a broker restart or crash,
does not replay messages after a destination comes back. Once a connector has taken a message from the queue, retrying a failed write is up to that connector type, and many simply log the failure,
does not acknowledge end to end: a publisher’s QoS 1/2 acknowledgement means the broker accepted the message, not that the destination stored it.

Filling the gaps

Upstream, use MQTT itself. Publish with QoS 1 or 2 from clients with persistent sessions (cleanSession=false, or a session expiry in MQTT 5), so the broker keeps their messages while they are offline. The broker’s durability settings decide what survives a broker restart.
Store first, forward later. Write each message to a database connector (or a model with STORE IN), then forward from that table on a schedule. The table is the durable buffer and can be replayed.
Let the destination buffer. Kafka, and a remote MQTT broker with persistent sessions, keep their own durable queues once a message reaches them.
Watch for drops. The broker logs when a connector starts dropping. Treat that as an alert.

When to use it

Leave the default (balanced) on most connectors.
Use lossless when a path must not lose messages (audit, compliance) and you accept that the publisher may wait.
Use realtime when high-rate telemetry must never stall the client that published it.
Set OVERLOAD_CAPACITY to cap one noisy connector so it cannot crowd out others.

ADD ROUTE_CONFIG is the only LoT block that works on every connector type. Place it next to that type’s own config (SQL_CONFIG, SOURCE_CONFIG, MODBUS_CONFIG, and so on). Order among ADD blocks does not matter.

Let’s look at how to set it, starting with the three common contracts.

Quick Start

Never drop a message
Never stall the publisher
Cap a noisy connector

Use lossless on a forwarding path that must keep every message. The publisher waits until this connector has room — it never drops.

DEFINE ROUTE CloudBridge WITH TYPE MQTT_BRIDGE
    ADD ROUTE_CONFIG
        WITH OVERLOAD_POLICY "lossless"
    ADD SOURCE_CONFIG
        WITH BROKER SELF
    ADD DESTINATION_CONFIG
        WITH BROKER_ADDRESS "broker.example.com"
        WITH BROKER_PORT '8883'
        WITH USE_TLS "true"
    ADD MAPPING auditForward
        WITH SOURCE_TOPIC "audit/#"
        WITH DESTINATION_TOPIC "cloud/audit/#"
        WITH DIRECTION "out"

Use realtime on high-rate telemetry. If the connector is full, the arriving message is dropped immediately and the publishing client is not delayed.

DEFINE ROUTE SensorDB WITH TYPE POSTGRESQL
    ADD ROUTE_CONFIG
        WITH OVERLOAD_POLICY "realtime"
    ADD SQL_CONFIG
        WITH SERVER GET ENV "PG_HOST"
        WITH DATABASE "factory"
        WITH USERNAME GET ENV "PG_USER"
        WITH PASSWORD GET SECRET "PG_PASSWORD"
    ADD EVENT StoreReading
        WITH SOURCE_TOPIC "sensors/+/reading"
        WITH QUERY "INSERT INTO readings (ts, value) VALUES (NOW(), '{value.json}')"

Cap how many of this connector’s messages may be in flight. Extra messages from this connector drop even while other connectors still have room.

DEFINE ROUTE NoisySensors WITH TYPE MQTT_BRIDGE
    ADD ROUTE_CONFIG
        WITH OVERLOAD_CAPACITY 5000
    ADD SOURCE_CONFIG
        WITH BROKER SELF
    ADD DESTINATION_CONFIG
        WITH BROKER_ADDRESS "telemetry.example.com"
        WITH BROKER_PORT '1883'
    ADD MAPPING sensorsOut
        WITH SOURCE_TOPIC "sensors/#"
        WITH DESTINATION_TOPIC "remote/sensors/#"
        WITH DIRECTION "out"

Settings

All three settings are optional. An absent ADD ROUTE_CONFIG block inherits the broker default entirely.

DEFINE ROUTE CloudBridge WITH TYPE MQTT_BRIDGE
    ADD ROUTE_CONFIG
        WITH OVERLOAD_POLICY "balanced"
        WITH OVERLOAD_TIMEOUT 200
        WITH OVERLOAD_CAPACITY 20000
    ADD SOURCE_CONFIG
        WITH BROKER SELF
    ADD DESTINATION_CONFIG
        WITH BROKER_ADDRESS "broker.example.com"
        WITH BROKER_PORT '8883'
        WITH USE_TLS "true"
    ADD MAPPING sensorSync
        WITH SOURCE_TOPIC "sensors/#"
        WITH DESTINATION_TOPIC "cloud/sensors/#"
        WITH DIRECTION "out"

Setting	Required	Default	Description
`OVERLOAD_POLICY`	No	`balanced`	`lossless` waits until there is room and never drops. `balanced` waits up to `OVERLOAD_TIMEOUT`, then drops the arriving message. `realtime` never waits — it drops immediately when the connector is full.
`OVERLOAD_TIMEOUT`	No	`50` (ms)	Wait budget for `balanced` only. Ignored for `lossless` (waits indefinitely) and `realtime` (never waits). Must be zero or greater.
`OVERLOAD_CAPACITY`	No	unbounded	Maximum number of this connector’s messages allowed in flight at once. Once reached, further messages from this connector drop even if other connectors still have room. Must be greater than zero.

A message already accepted for a connector is never evicted. Only the arriving message can be dropped. That keeps sequences such as Sparkplug birth then data self-consistent under load.

Do not set OVERLOAD_CAPACITY together with OVERLOAD_POLICY "lossless". A capacity cap drops overflow; lossless never drops. The broker logs an error naming the connector, ignores the capacity, and runs the connector as lossless. The same conflict applies if the connector inherits lossless from the broker default and then declares a capacity.

Defaults

Each setting resolves on its own:

No ADD ROUTE_CONFIG block — the connector inherits the broker default for policy, timeout, and capacity.
A block is present but a setting is omitted — that setting does not copy the broker’s timeout or capacity. OVERLOAD_POLICY inherits the broker default. OVERLOAD_TIMEOUT and OVERLOAD_CAPACITY use that policy’s own default (50 ms for balanced, unbounded capacity).
An invalid value (unknown policy name, negative timeout, non-positive capacity) does not stop the connector loading. The broker logs an error and uses that field’s default.

Broker-wide defaults are environment variables, read once at process start. Change them and restart the broker. A connector’s own ADD ROUTE_CONFIG always overrides these for that connector.

Variable	Default	Effect
`COREFLUX_ROUTE_OVERLOAD_POLICY`	`balanced`	Default for `OVERLOAD_POLICY` (`lossless`, `balanced`, or `realtime`). An unrecognized value logs an error and falls back to `balanced`.
`COREFLUX_ROUTE_CHANNEL_FULL_TIMEOUT_MS`	`50`	Default wait budget in milliseconds for `balanced`. Ignored when the resolved policy is `lossless` or `realtime`. A negative value logs an error and falls back to `50`.
`COREFLUX_ROUTE_CHANNEL_CAPACITY`	`100000`	Shared ceiling on how many connector updates the broker will hold at once. Below the ceiling, matching messages are accepted. At the ceiling, the arriving message’s wait-or-drop policy applies. Values below `1024` are raised to `1024`. Unset, non-numeric, zero, or negative values fall back to `100000`.

OVERLOAD_CAPACITY on a connector is a private budget for that connector. The shared ceiling is separate: one connector can hit its own cap and start dropping while other connectors still have room. See Broker Configuration for how these sit alongside other broker limits.

What happens under load

When a connector is waiting, that wait can delay the publishing client’s own publish — only that client, and only up to the wait budget. Other connected clients are not stalled by this connector. The broker logs two different saturation conditions (each at most once every five seconds):

The shared connector queue is full — every connector waiting behind it is affected.
A single connector is over the OVERLOAD_CAPACITY it declared for itself — the shared queue may still have room.

If two connectors could match the same topic (for example one exact mapping and another sensors/#), the broker applies the stricter wait-or-drop choice among those candidates. A lossless connector on an overlapping pattern can make a realtime connector wait, even if only the realtime connector would have handled the message. The broker never drops a message because a more lenient connector asked to drop it. If mixed policies on overlapping wildcards wait or drop more than you expect, check the patterns first — that behavior is by design.

Troubleshooting

A connector drops messages even though the broker looks idle

That connector’s own OVERLOAD_CAPACITY is exhausted. Look for the broker warning that a connector is over its in-flight budget, not the warning that the shared queue is full.

Publishers on a lossless connector stall under load

Working as designed — lossless waits until there is room. Switch to balanced with a timeout if you would rather drop than stall, or cap a noisy connector with OVERLOAD_CAPACITY so it drops its own overflow instead.

OVERLOAD_CAPACITY seems to have no effect

It is declared alongside OVERLOAD_POLICY "lossless" (or the broker default is lossless). Check the broker log for the lossless/capacity conflict — capacity is ignored, not applied.

A connector with no wildcard still waits longer than its own policy

Another connector’s overlapping wildcard pattern resolved a stricter policy for that topic. See the overlapping-pattern warning above.

The connector drops immediately and never waits

The resolved policy is realtime. That is expected — realtime drops without waiting.

Best practices

Start with the default balanced policy. Change a connector only when you have a reason (must not lose, must not stall, or must not crowd others).
Use lossless sparingly. It protects delivery at the cost of delaying the publisher that caused the backlog.
Put OVERLOAD_CAPACITY on the noisy connector, not on the critical one — the critical connector can stay unbounded while the noisy connector drops its own overflow.
Prefer exact topic mappings when two connectors would otherwise share a broad + / # pattern with different policies.
Invalid settings never fail the connector silently: read the broker log after deploy if behavior does not match the block you wrote.

Quick Start

Coreflux MQTT Broker

LoT Language

Coreflux HUB

VS Code - LoT Notebooks

Coreflux Portal

Using AI with Coreflux

Product Updates

Store and forward in LoT connectors

The principles, and how LoT handles them

What it does not do

Filling the gaps

When to use it

Quick Start

Settings

Defaults

What happens under load

Troubleshooting

Best practices

Next Steps

Connectors Overview

Broker Configuration

​Store and forward in LoT connectors

​The principles, and how LoT handles them

​What it does not do

​Filling the gaps

​When to use it

​Quick Start

​Settings

​Defaults

​What happens under load

​Troubleshooting

​Best practices

​Next Steps

Connectors Overview

Broker Configuration

Store and forward in LoT connectors

The principles, and how LoT handles them

What it does not do

Filling the gaps

When to use it

Quick Start

Settings

Defaults

What happens under load

Troubleshooting

Best practices

Next Steps