Skip to content

File Storage Route

Feature Since Version Notes
File Storage Route >v1.6.3 Stores MQTT messages in the local filesystem.

1. Overview

The File Storage Route provides a mechanism to store MQTT message payloads directly into the local filesystem. This route is highly configurable, supporting different file formats, naming conventions, and storage strategies.

For practical examples of LOT Routes, check out the Routes examples in the LOT Samples Repository. It is an excellent choice for simple data logging, debugging, creating data archives, or for scenarios where an external database is not required.

2. Key Features

  • Multiple Formats: Supports storing data as either JSON or CSV files.
  • Customizable File Naming: Define a file name pattern that can include the target "table" (topic or other identifier).
  • Append or Overwrite: Configure whether to append data to existing files or create new ones.
  • Data Rotation: For JSON files, limits the number of records per file, providing a simple rotation mechanism.
  • Timestamping: Optionally adds a timestamp to each stored record.
  • Thread-Safe: Ensures that concurrent writes to the same file are handled safely.

3. Route Configuration

A route can be configured using the addRoute command with TYPE FILE_STORAGE.

3.1 Example: CSV File Storage

This example configures a route to store data in CSV files within the csv_logs directory. 

DEFINE ROUTE CSVLogger WITH TYPE FILE_STORAGE
    ADD FILE_STORAGE_CONFIG
        WITH STORAGE_DIR "csv_logs"
        WITH FORMAT "CSV"
        WITH FILE_PATTERN "{0}.csv"
        WITH APPEND TRUE
        WITH INCLUDE_TIMESTAMP TRUE
        WITH CSV_DELIMITER ","

3.2 Example: JSON File Storage with Rotation

This example configures a route to store data as JSON, keeping only the latest 5000 records per file. 

DEFINE ROUTE JSONLogger WITH TYPE FILE_STORAGE
    ADD FILE_STORAGE_CONFIG
        WITH STORAGE_DIR "json_logs"
        WITH FORMAT "JSON"
        WITH FILE_PATTERN "{0}.json"
        WITH APPEND TRUE
        WITH INCLUDE_TIMESTAMP TRUE
        WITH MAX_RECORDS 5000

Adding Routes via MQTT Command

To deploy these routes to the broker, use the -addRoute command by publishing to the $SYS/Coreflux/Command topic:

-addRoute DEFINE ROUTE CSVLogger WITH TYPE FILE_STORAGE
    ADD FILE_STORAGE_CONFIG
        WITH STORAGE_DIR "csv_logs"
        WITH FORMAT "CSV"
        WITH FILE_PATTERN "{0}.csv"
        WITH APPEND TRUE
        WITH INCLUDE_TIMESTAMP TRUE
        WITH CSV_DELIMITER ","

For more information about broker commands, see the MQTT Broker Commands documentation.

Deploying Routes with LOT Notebooks

Routes can also be created and deployed interactively using LOT Notebooks in Visual Studio Code. This provides a convenient way to develop, test, and document your routes.

To get started with LOT Notebooks:

  1. Install the LOT Notebooks extension in VS Code

  2. Create a new .lotnb file

  3. Add code cells with your route definitions

  4. Execute the cells to deploy directly to your broker

For detailed instructions, see the LOT Notebooks Getting Started Guide.

3.3 Explanation of Configuration Options

The configuration is provided within a FILE_STORAGE_CONFIG block. Note: The legacy SOURCE_CONFIG name is also accepted for this block.

General Configuration

  • STORAGE_DIR: The directory where files will be stored. Defaults to data.
  • FORMAT: The file format. Can be JSON or CSV. Defaults to JSON.
  • FILE_PATTERN: A pattern for the filename. Use {0} as a placeholder for the targetName specified in the STORE action. Defaults to {0}.json or {0}.csv.
  • APPEND: Determines if data is appended to an existing file. For JSON, records are added to a JSON array. For CSV, new rows are added. Can be true or false. Defaults to true.

Format-Specific Configuration

  • INCLUDE_TIMESTAMP: If true, a timestamp field is added to each record. Defaults to true.
  • DATE_FORMAT: The format string for the timestamp (e.g., yyyy-MM-dd HH:mm : ss).
  • CSV_DELIMITER: The character to use as a delimiter in CSV files. Defaults to ,.
  • MAX_RECORDS: For JSON files, this sets the maximum number of records to keep in the file's array. Older records are discarded to maintain the limit. Defaults to 10000.

4. Data Storage and File Handling

4.1 Usage in Actions

The File Storage route is used with a STORE action. The targetName in the action is substituted into the {0} placeholder in the FILE_PATTERN. Apply to RouteHandler...

4.2 File Formats

  • JSON: Data is stored as an array of JSON objects. If APPEND is true, new objects are added to the array in the target file.
  • CSV: The first line of the file will be a header row derived from the JSON keys of the first message stored. Subsequent messages are added as new rows. Nested JSON is flattened with . as a separator.

5. Using Models for Structured File Content

While the File Storage route can save raw message payloads, its utility is greatly enhanced when used with LOT Models. Models allow you to define a consistent schema, transform incoming data, and enrich it with additional information before it's written to a file.

By using a Model, you can ensure that your log files (whether JSON or CSV) have a predictable structure. The STORE IN command within the model definition directs the processed data to a File Storage route, which then handles writing it to the filesystem.

Example: Creating a Structured JSON Log

This example shows how to use a model to process sensor data and save it as a structured entry in a JSON file.

1. The Model Definition

DEFINE MODEL MyFileLogModel COLLAPSED WITH TOPIC "MyFileLogModel"
    ADD "event_ts" WITH TIMESTAMP "ISO"
    ADD "device_id" WITH TOPIC "sensors/+/id"
    ADD "temperature" WITH TOPIC "sensors/+/temp" AS TRIGGER
    STORE IN "MyFileLogger"
        WITH TABLE "sensor_logs"
How it works: - DEFINE MODEL ...: Creates a model to structure log entries. - ... AS TRIGGER: The model runs whenever a new temperature reading is published. - ADD ...: The model captures the device ID and adds a standard ISO timestamp. - STORE IN "MyFileLogger" ...: This sends the final JSON object to the MyFileLogger route. The TABLE name sensor_logs is used as the placeholder in the route's FILE_PATTERN (e.g., {0}.json becomes sensor_logs.json).

2. The Resulting JSON File Content

If the MyFileLogger route is configured for JSON format, each trigger of the model would append a new object to the array in sensor_logs.json: 

[
  {
    "event_ts": "2023-03-15T12:00:00Z",
    "device_id": "temp-sensor-01",
    "temperature": 22.5
  },
  {
    "event_ts": "2023-03-15T12:01:00Z",
    "device_id": "temp-sensor-01",
    "temperature": 22.6
  }
]