Skip to content

LOT Language - Model Management Documentation

1. Overview

Models in Coreflux use the LOT language syntax to define how data is processed, transformed, and published. Models take input data (triggered by specific topics), process it through expressions, constants, or transformations, and output the results to new MQTT topics.

For practical examples of LOT Models, check out the Models examples in the LOT Samples Repository.

Models serve as data pipelines or virtual sensors that can enrich data from existing MQTT topics, perform computations, and publish processed results. By using LOT language models, you can easily configure and maintain logic at the broker level, enabling dynamic calculation and formatting of data without the need for external services.

1.1 Two Types of Models

Coreflux supports two distinct types of models:

  1. Data Transformation Models: Traditional models that process and transform data in real-time
  2. LOT Models: Structured data schemas used with PUBLISH MODEL commands in Actions

2. LOT Models (Structured Data Schemas)

LOT Models provide a powerful way to define structured data schemas within Coreflux. They are used in combination with the PUBLISH MODEL command inside ACTIONs to create and send well-formed, predictable JSON payloads without the need for manual and error-prone string construction.

This approach promotes reusability, improves readability, and ensures data consistency across your system.

2.1 DEFINE MODEL for Structured Data

The DEFINE MODEL statement is used to declare a reusable data structure, or schema.

Syntax

DEFINE MODEL <ModelName> [COLLAPSED]
    ADD <TYPE> "<FieldName>"
    ADD COLLECTION "<CollectionName>" OF <AnotherModelName>
  • <ModelName>: A unique name for your data model.
  • [COLLAPSED]: An optional keyword that creates a flat key-value JSON object. If omitted, the model will produce a nested JSON object with the model's name as the root key.
  • ADD <TYPE> "<FieldName>": Defines a field within the model.

Supported Field Types

Type Description
STRING A text string.
INT An integer number.
DOUBLE A floating-point number.
BOOL A boolean value (true or false).
OBJECT A raw JSON object or array. The value provided in PUBLISH MODEL will be parsed and embedded directly, not as a string. This is extremely useful for complex, nested structures or when part of the payload is dynamic.
COLLECTION Represents a JSON array containing objects of another defined model type.

2.2 PUBLISH MODEL Command

The PUBLISH MODEL command is used within an ACTION to instantiate a defined model with data and publish the resulting JSON payload to an MQTT topic.

Syntax

PUBLISH MODEL <ModelName> TO "<topic>" WITH
    FieldName1 = <value1>
    FieldName2 = <value2>
    ...
  • <ModelName>: The name of the DEFINE MODEL to use.
  • **"<topic>"**: The MQTT topic to publish the final JSON payload to.
  • FieldName = <value>: Assigns a value to a field defined in the model. Values can be literals, variables from SET commands, or results from functions like GET JSON.

2.3 LOT Model Examples

Example 1: Simple, Collapsed Model

DEFINE MODEL KafkaMessage COLLAPSED
    ADD STRING "componentId"
    ADD STRING "topic"
    ADD OBJECT "payload"

DEFINE ACTION ProcessComponentsPayload
ON TOPIC "Raw/Components/+/+" DO
    SET "systemId" WITH (GET JSON "info" IN PAYLOAD AS STRING)
    SET "topicv" WITH ("Components/" + TOPIC POSITION 3 + "/" + TOPIC POSITION 4 + "/" + {systemId})

    PUBLISH MODEL KafkaMessage TO ({topicv}+"/Unit1") WITH
        componentId = (TOPIC POSITION 3 + "_" + TOPIC POSITION 4 + "_" + {systemId} + "_" + "Unit1")
        topic = ({topicv}+"/Unit1")
        payload = (GET JSON "Unit1" IN PAYLOAD AS STRING)

Example 2: Complex, Nested Model

DEFINE MODEL SmartLight COLLAPSED
    ADD STRING "deviceId"
    ADD BOOL "online"
    ADD OBJECT "state"
        ADD STRING "power"
        ADD INT "brightness"    // 0-100
        ADD INT "temperature"   // Color temperature in Kelvin
    ADD TIMESTAMP "last_updated" WITH "UNIX"

DEFINE ACTION UpdateLightState
ON TOPIC "lights/office/status" DO
    PUBLISH MODEL SmartLight TO "lights/office/state" WITH
        deviceId = "light-office-1",
        online = true,
        state.power = (GET JSON "power" IN PAYLOAD AS STRING),
        state.brightness = (GET JSON "brightness" IN PAYLOAD AS INT),
        state.temperature = (GET JSON "temp" IN PAYLOAD AS INT)

2.4 STORE Functionality

LOT Models can be integrated with database storage using the STORE clause and ROUTEs.

Database Routes

DEFINE ROUTE mongo_route WITH TYPE MONGODB
    ADD MONGODB_CONFIG
        WITH CONNECTION_STRING "mongodb+srv://<username>:<password>@<cluster-uri>/<database>?tls=true&authSource=admin&replicaSet=<replica-set>"
        WITH DATABASE "admin"

Model with Storage

DEFINE MODEL MachineData WITH TOPIC "Simulator/Machine/+/Data"
    ADD "energy" WITH TOPIC "raw_data/+" AS TRIGGER
    ADD "device_name" WITH REPLACE "+" WITH TOPIC POSITION 2 IN "+"
    ADD "energy_wh" WITH (energy * 1000)
    ADD "production_status" WITH (IF energy > 5 THEN "active" ELSE "inactive")
    ADD "production_count" WITH (IF production_status EQUALS "active" THEN (production_count + 1) ELSE 0)
    ADD "stoppage" WITH (IF production_status EQUALS "inactive" THEN 1 ELSE 0)
    ADD "maintenance_alert" WITH (IF energy > 50 THEN TRUE ELSE FALSE)
    ADD "timestamp" WITH TIMESTAMP "UTC"
    STORE IN "mongo_route"
        WITH TABLE "MachineProductionData"

3. Data Transformation Models (Traditional)

3.1 Model Processing Overview

Models operate as functions or pipelines triggered whenever their input topics receive new messages. When a trigger topic updates:

  1. Input Acquisition: The model reads the defined input properties. Some properties may be constants, others may reference topics.
  2. Computation and Expressions: Each property's definition can involve direct values, references to other properties, or arithmetic/conditional expressions.
  3. Output Publishing: After processing, the model publishes each property's current value to the model's output namespace, essentially creating a set of derived topics.

Models can thus aggregate or transform data from multiple inputs and produce consolidated or computed outputs.

3.2 Properties in Models

Each model consists of one or more properties defined with the ADD statement. A property can be:

  • Directly from a Topic (with AS TRIGGER or without): If AS TRIGGER is used, changes in that topic will cause the model to recompute and update outputs.
  • A Fixed Constant: A property can be assigned a static number or string.
  • An Expression: A property can be computed from other properties using arithmetic operations, conditional expressions (IF ... THEN ... ELSE), or timestamps.

3.3 Topics and Unified Namespace

Models can use wildcards (+) in their topics to dynamically handle multiple sources in a Unified Namespace (UNS). For example, if a model's output topic is Coreflux/+/+/+/+/energy, and it references an input topic shellies/+/+/+/+/device/energy as a trigger, then any shellies/x/y/z/w/device/energy incoming value will map to Coreflux/x/y/z/w/energy/... outputs.

3.4 Constants and Expressions

Properties can hold constant values or can be computed using arithmetic operations, conditional logic, or timestamps. For example:

  • Constant: ADD "energy_price" WITH 3
  • Expression: ADD "cost" WITH (total_energy * energy_price)

These capabilities allow building complex logic directly in the broker without external code.

4. Default Behavior

By default, models do not apply any transformations unless defined. They do not alter the original incoming data but produce new topics for processed results. If a model references a topic that never updates, the model will not publish new outputs.

5. Model Syntax

Models in the LOT language follow this general syntax:

DEFINE MODEL <model_name> WITH TOPIC "<output_base_topic>"
    ADD "<property_name>" WITH TOPIC "<input_topic>" [AS TRIGGER]
    ADD "<property_name>" WITH <constant_value>
    ADD "<property_name>" WITH (expression)

5.1 Components Explained

  • DEFINE MODEL <model_name>: Defines a new model with a unique <model_name>.
  • WITH TOPIC <output_base_topic>: Specifies the base output topic for the model's computed properties.
  • ADD <property_name> ...: Defines a property of the model.
    • WITH TOPIC "" [AS TRIGGER]: Indicates this property's value comes from another MQTT topic. If AS TRIGGER is present, updates on this topic cause the model to recompute.
    • WITH : Assigns a fixed constant value to the property.
    • WITH (expression): Defines a property's value as an expression based on other properties, arithmetic, conditions, or timestamps.

5.2 Timestamps and Conditions

  • IF Conditions:

    ADD "message" WITH IF alarm == "true" THEN "all ok!" ELSE "message has an alarm 33"

  • Timestamp:

    ADD "timestamp" WITH TIMESTAMP "UTC"

    This can add a timestamp to your computed data in various formats ("UTC", "ISO", "UNIX-MS", or "UNIX").

6. Model Management Guidelines

Like rules, models can be added or removed dynamically using commands published to the $SYS/Coreflux/Command topic.

6.1 Add a Model: addModel <DEFINE MODEL code>

This command allows you to add a new model to the system. Example:

-addModel DEFINE MODEL LampEnergyCost WITH TOPIC "Coreflux/+/+/+/+/energy"
    ADD "total_energy" WITH TOPIC "shellies/+/+/+/+/device/energy" AS TRIGGER
    ADD "energy_price" WITH 3
    ADD "cost" WITH (total_energy * energy_price)

In this example:

  • The model is named LampEnergyCost.
  • The output base topic is Coreflux/+/+/+/+/energy.
  • total_energy comes from shellies/+/+/+/+/device/energy and triggers computations.
  • energy_price is a constant 3.
  • cost is computed as (total_energy * energy_price).

Whenever a shellies/.../device/energy message arrives, the model computes outputs and publishes to Coreflux/.../energy/total_energy, Coreflux/.../energy/energy_price, and Coreflux/.../energy/cost.

Another example:

-addModel DEFINE MODEL LampEnergyCostFixed WITH TOPIC "shellies/Coreflux/Students/Lamp/Lamp1/costs"
    ADD "total_energy" WITH TOPIC "shellies/Coreflux/Students/Lamp/Lamp1/device/energy" AS TRIGGER
    ADD "energy_price" WITH 2
    ADD "cost" WITH (total_energy * energy_price)

Here, the model is fixed to a specific device path without wildcards, outputting computed values to shellies/Coreflux/Students/Lamp/Lamp1/costs/....

6.2 Remove a Model: removeModel <model_name>

This command removes an existing model by specifying its name. Example:

-removeModel LampEnergyCost

In this example, the model LampEnergyCost is removed from the system. This stops the model from processing future messages and publishing outputs.

7. Additional Examples

7.1 UNS Style Models

DEFINE MODEL GenericEnergyCost WITH TOPIC "Coreflux/+/+/+/+/energy"
    ADD "total_energy" WITH TOPIC "shellies/+/+/+/+/device/energy" AS TRIGGER
    ADD "energy_price" WITH 3
    ADD "cost" WITH (total_energy * energy_price)

For any device matching the shellies/.../device/energy pattern, this model will process data and publish to Coreflux/.../energy/....

7.2 Direct, Singleton Model

DEFINE MODEL SpecificMachineStopOrder WITH TOPIC "company/factory/department/machine/stopOrder"

If a JSON payload like:

{
  "reason": 42,
  "stop": true,
  "time": "2024-12-11T15:12:54Z"
}

arrives at company/factory/department/machine/stopOrder, the model will break it down into:

  • company/factory/department/machine/stopOrder/reason -> 42
  • company/factory/department/machine/stopOrder/stop -> true
  • company/factory/department/machine/stopOrder/time -> "2024-12-11T15:12:54Z"

(Note: If you want to define additional properties or triggers, you can ADD them similarly.)

7.3 Conditional Message Models

DEFINE MODEL AlertMessage2 WITH TOPIC "Company2/+/+/+/AlarmMessage"
    ADD "alarm" WITH TOPIC "shellies2/+/+/+/device/alarm" AS TRIGGER
    ADD "message" WITH IF alarm == "true" THEN "all ok!" ELSE "message has an alarm 33"
    ADD "timestamp" WITH TIMESTAMP "UTC"

When alarm triggers, the model publishes a message to Company2/.../AlarmMessage/... including a processed message and a timestamp.

7.4 Counters and Running Totals

DEFINE MODEL ProductionCounterModel WITH TOPIC "machine/productionCounter"
    ADD "Value" WITH TOPIC "machine/signal/countPart" AS TRIGGER
    ADD "Counter" WITH (Counter + Value)

Each time machine/signal/countPart receives a new number, the model adds it to the Counter property, creating a running total published under machine/productionCounter/Counter.

7.5 Advanced Data Transformation Models with Storage

Sensor Data Aggregation Model

DEFINE MODEL SensorAggregation WITH TOPIC "Sensors/+/Aggregated"
    ADD "temperature" WITH TOPIC "sensor/+/temperature" AS TRIGGER
    ADD "humidity" WITH TOPIC "sensor/+/humidity" AS TRIGGER
    ADD "pressure" WITH TOPIC "sensor/+/pressure" AS TRIGGER
    ADD "sensor_id" WITH REPLACE "+" WITH TOPIC POSITION 2 IN "+"
    ADD "temp_fahrenheit" WITH ((temperature * 9/5) + 32)
    ADD "comfort_index" WITH ((temp_fahrenheit + humidity) / 2)
    ADD "alert_level" WITH (IF comfort_index > 80 THEN "high" ELSE IF comfort_index > 60 THEN "medium" ELSE "low")
    ADD "timestamp" WITH TIMESTAMP "UTC"
    STORE IN "postgres_route"
        WITH TABLE "SensorReadings"

IoT Device Status Model

DEFINE MODEL DeviceStatus WITH TOPIC "IoT/+/Status"
    ADD "voltage" WITH TOPIC "device/+/voltage" AS TRIGGER
    ADD "current" WITH TOPIC "device/+/current" AS TRIGGER
    ADD "device_id" WITH REPLACE "+" WITH TOPIC POSITION 2 IN "+"
    ADD "power_watts" WITH (voltage * current)
    ADD "status" WITH (IF power_watts > 0 THEN "online" ELSE "offline")
    ADD "health_score" WITH (IF voltage > 10 AND current > 0.1 THEN 100 ELSE 50)
    ADD "last_seen" WITH TIMESTAMP "UTC"
    STORE IN "mysql_route"
        WITH TABLE "DeviceHealth"

8. Best Practices

8.1 LOT Models vs Data Transformation Models

  • Use LOT Models when you need structured, predictable JSON output for external APIs or systems
  • Use Data Transformation Models when you need real-time data processing and transformation
  • Combine both for complex scenarios requiring both structured output and real-time processing

8.2 Database Integration

  • Choose the appropriate database type for your use case:
  • MongoDB: Document-based storage for flexible data structures
  • PostgreSQL: Relational database for structured sensor data
  • MySQL: Traditional relational database for device management
  • InfluxDB: Time-series database for environmental metrics
  • TimescaleDB: Time-series extension of PostgreSQL for energy data

8.3 Performance Considerations

  • Use COLLAPSED models for flat JSON structures when possible
  • Leverage wildcards (+) for scalable multi-device processing
  • Use OBJECT type for complex, dynamic JSON structures
  • Implement proper error handling in your models